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Abstract 

Expert  system  technology  appears  to  hold  considerable 
promise  for  enhancing  productivity  and  promoting  better 
decision-making.  The  purpose  of  this  study  was  to  determine 
if  an  expert  system  application  for  asset  reconciliation 
could  improve  inventory  management  procedures  and  poten¬ 
tially  produce  savings  for  the  Government  while  alleviating 
manpower  shortages. 

The  primary  objective  of  this  research  was  to  validate 
and  extend  a  previous  research  finding  that  documented  a  15 
percent  increase  in  the  effectiveness  of  inventory  managers 
when  assisted  by  an  expert  system.  Research  was  conducted 
to  develop  an  expert  system  for  a  different  task  and  to 
measure  the  performance  level  of  this  expert  system,  both  in 
terms  of  the  effectiveness  and  the  efficiency  of  item 
managers  who  were  assisted  by  the  system. 

The  methodological  approach  to  the  research  effort 
involved  a  twelve-step  process  that  was  divided  into  three 
general  phases.  The  objective  of  the  first  phase,  problem 
selection,  was  to  choose  a  problem  to  be  addressed  by  the 
expert  system  and  to  select  the  domain  experts  whose 
knowledge  would  be  encoded  into  the  expert  system  to  solve 
the  chosen  problem.  The  objective  of  the  second  phase, 
knowledge  engineering,  was  to  extract  the  knowledge  from  the 


selected  experts  in  order  to  create  a  knowledge  base  for  the 
expert  system.  The  objective  of  the  final  phase, 
performance  evaluation,  was  to  determine  if  the  developed 
expert  system  enhanced  or  detracted  from  the  productivity  of 
inventory  managers  using  the  system. 

The  expert  system  developed  was  titled  the  Manager’s 
Asset  Reconciliation  System  (MARS).  MARS  improved  the 
overall  effectiveness  of  item  managers,  who  used  the  system 
to  complete  test  cases,  by  19.45  percent.  There  appeared  to 
be  a  reduction  in  the  efficiency  of  item  managers  assisted 
by  the  expert  system.  The  lack  of  user  familiarity  with 
computer  operations  and  MARS  was  assumed  to  contribute  to 
this  reduction.  This  study  also  provides  suggestions  for 
modifications  to  MARS  that  will  improve  the  expert  system 
consultation  thereby  increasing  the  efficiency  of  the 
system.  User  acceptance  of  MARS  was  high  and 
recommendations  for  further  testing  of  MARS  are  provided. 


An  Expert  System  for  Asset  Reconciliation 

I .  Introduction 

Overv 1 ew 

The  technology  of  Artificial  Intelligence  (AI), 
specifically  expert  systems,  appears  to  have  considerable 
capability  for  enhancing  productivity  and  promoting  better 
decision-making.  The  technology  has  emerged  from  the 
research  laboratory  demonstrating  successful  applications 
for  a  wide  range  of  business  operations.  A  1986  article  in 
Dun's  Bus iness  Month.  "Computers  Think  for  Business," 
states : 

Companies  in  virtually  every  industry  are 
using  expert  systems  and  making  major  efforts 
to  disseminate  the  technology  throughout 
their  organizations.  Close  to  100  custom- 
built  systems  and  more  than  1000  off-the- 
shelf  systems  are  now  in  operation  [21:30]. 

The  potential  power  of  expert  systems  to  replicate  rare 
or  expensive  human  expertise  has  led  to  a  worldwide  effort 
to  extend  and  apply  this  technology  [51:vu  ].  Harmon  and 
King  note  in  their  book  Expert  Systems:  Artificial  Intelli¬ 
gence  f or  Bus iness: 

Today,  the  United  States,  Japan,  England,  and 
the  European  Economic  Community  are  ah  in 
the  process  of  launching  major  research 
programs  to  develop  and  implement  expert 
systems  in  the  near  future  [23:1], 


The  Air  Force  Logistics  Command  (AFLC)  has  recognized 
the  potential  of  AI  and  has  developed  a  strategy  for  apply¬ 
ing  expert  system  technology  to  the  command’s  maintenance, 
acquisition,  requirements,  and  distribution  activities. 
Expanding  workloads,  serious  shortages  in  expertise,  and 
'belt  tightening*  budgets  make  such  a  strategy  crucial  to 
needed  improvements  in  productivity.  To  quote  the  CORONA 
REQUIRE  study  which  was  chaired  by  General  Alton  D.  Slay, 
USAF  (Retired)  and  presented  to  the  Chief  of  Staff  and  the 
Secretary  of  the  Air  Force  in  1983: 

Given  the  n o n - r e s po n s i v e  Automated  Data 
Processing  (ADP) .  there  are  insufficient 
numbers  of  Item  M a n ag e me n t / Eq u i p me n t 
Management  and  Contracting  Specialists. 
Manning  levels  for  each  have  dropped 
from  around  90%  to  the  70-75%  range  in  the 
last  two  years  while  the  workload  has  almost 
doubled.  Further,  the  skill  level,  particu¬ 
larly  of  the  Item  Management  Specialists,  is 
a  serious  problem  as  rapid  turnover  has 
drained  the  experience  base  [43:7]. 

Until  recently.  Air  Force  AI  applications  primarily 
addressed  the  problems  of  battle  management,  design  engine¬ 
ering,  and  the  cockpit  environment,  with  relatively  little 
emphasis  on  the  area  of  inventory  management.  Expert  sys¬ 
tems,  however,  offer  tremendous  potential  for  improving  the 
management  of  inventory  [3:24].  Poor  inventory  management 
wastes  valued  resources.  AFLC  capital  assets  equalled 
*115.7  billion  as  of  March  1987  [15:2-39].  In  addition,  the 
Command  bought,  stored,  issued  and  distributed  929,700  items 
in  fiscal  year  1986  [14:iii],  Even  a  small  improvement  in 
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the  management  of  these  items  suggests  significant  savings 
could  be  realized  by  AFLC  and  the  Federal  Government. 

The  objective  of  this  research  was  to  validate  a  pre¬ 
vious  research  finding  that  documented  a  15%  increase  in  the 
effectiveness  of  inventory  managers  when  assisted  by  an 
expert  system.  Further,  the  research  discussed  herein  ex¬ 
tended  the  previous  study  to  an  additional  area  of  inventory 
management,  specifically,  the  area  of  asset  reconciliation. 
Background  information  on  expert  systems  is  provided  next  so 
the  reader  may  better  understand  the  proposed  research  which 
was  conducted.  Basic  expert  system  terminology  is  defined 
in  Appendix  A. 


Background 


Expert  Systems  are  computer  software  programs  that 


emulate  limited  aspects  of  human  thinking.  An  expert  system 
can  be  defined  as,  ‘any  system  that  can  process  data  and 
deliver  or  administer  the  advice  or  expertise  a  human  expert 
would  give  under  the  same  circumstances  [38:58].' 

Expert  systems  execute  chains  of  logical  reasoning 
linking  facts  and  conclusions  by  incorporating  human  know¬ 
ledge  and  experience  [32:35].  Expert  systems  use  search 
heuristics,  described  as  'procedures,  strategies,  and  rules 
of  thumb  for  problem  solving  [51:5],'  to  reduce  the  size  of 
the  problem  space  and  to  arrive  at  a  solution.  The  human 
expert’s  heuristics  are  loaded  into  the  expert  system’s 
'knowledge  base'  as  rules  and  facts  about  the  area  or  domain 
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of  interest  from  which  decisions  can  be  derived- 

The  most  common  method  used  to  represent  knowledge  in 

contemporary  AI  applications  is  the  rule-based  method 
[41:93].  Knowledge  is  represented  as  "a  collection  of  indi¬ 
vidual  situation-action  rules,  each  one  of  which  captures  a 
single  inference,  action,  or  contingency  for  a  particular 
class  of  situations  [41:93].'  An  example  of  a  rule  for  a 
medical  diagnosis  expert  system  might  be  represented  as 
follows : 

if  the  patient’s  blood  pressure  is  dropping 
and  there  is  no  internal  injury 
then  suspect  internal  bleeding  [41:93]." 

The  term  "expert"  is  described  by  Paul  E.  Johnson,  a 

behavioral  scientist,  in  the  following  quote: 

A  person  who,  because  of  training  and  experience, 
is  able  to  do  things  the  rest  of  us  cannot; 
experts  are  not  only  proficient  but  also  smooth 
and  efficient  in  the  actions  they  take.  Experts 
know  a  great  many  things  and  have  tricks  and 
caveats  for  applying  what  they  know  to  problems 
and  tasks  [51:5]. 

The  inference  engine,  the  system's  data  manager  and 
query  processor,  uses  a  search  strategy  that  directs  the 
expert  system  to  evaluate  only  those  rules  and  facts  that 
are  applicable  to  the  problem  at  hand.  Each  time  the  system 
is  queried  by  the  user  a  response  is  made  by  searching 
through  the  rules  and  facts  in  the  system's  knowledge  base. 
If  the  system  is  unable  to  reach  a  conclusion  based  on 
present  knowledge  of  the  situation,  the  user  is  prompted  to 
supply  additional  information.  This  information  can  be 


added  to  other  conclusions  stored  during  the  consultation 


with  the  expert  system,  thereby  upgrading  the  system.  The 
process  continues  until  the  system  reaches  a  final  conclu¬ 
sion  and  recommends  a  management  action  [10;47:3]. 

Figure  1  describes  the  basic  structure  of  an  expert 
system.  The  actual  process  of  building  an  expert  system  is 
called  knowledge  engineering  [51:5].  The  building  process 
is  conducted  by  a  knowledge  engineer  whose  job  involves 
determining  what  an  expert  does  when  making  decisions  and 
organizing  this  information  into  an  effective  expert  system 
implementation  [22:13].  Research  has  determined  that  domain 
practitioners,  i.e.  logisticians,  make  the  best  knowledge 
engineers  for  their  domain  [5:6].  Knowledge  acquisition 
facilities  are  used  to  transfer  the  knowledge  from  the  human 
expert  or  other  information  source  into  a  form  that  can  be 
used  by  the  expert  system  [45:G-10],  The  knowledge  base 
contains  information  gleaned  from  the  human  expert  by  the 
knowledge  engineer  [25:10].  The  inference  system  provides 
overall  control  of  the  system  and  acts  as  an  explanation 
generator,  supplying  information  to  the  user  that  describes 
how  the  expert  system  arrived  at  solutions  [3:18-20]. 

Expert  systems  can  be  applied  to  the  general  classes  of 
problem  solving  shown  in  Figure  2.  Problems  that  involve 
the  manipulation  of  large  quantities  of  information  using 
heuristics,  pattern  matching,  and  searching  procedures  are 
considered  well  suited  for  expert  system  application.  The 
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Figure  1 

Basic  Structure  of  an  Expert  System  [18:76] 
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f  Expert  System  Problem  Solving  [23:3] 


7 


1 


development  of  any  expert  system  requires  the  existence  of  a 
person  who  can  already  successfully  solve  the  problem. 
Ideally  this  human  expert  is  available  and  committed  to  the 
development  of  an  expert  system  for  the  problem  domain. 

Problems  which  make  good  candidates  for  an  expert  sys¬ 
tem  are  those  problems  where  knowledge  and  experience  have  a 
significant  effect  on  the  accuracy  of  the  solution  chosen 
[46:11].  If  this  knowledge  is  crucial,  yet  concentrated  in 
just  a  few  individuals,  the  development  of  an  expert  system 
is  especially  suitable  for  an  organization.  The  completed 
s j  'tern  should  offer  a  significant  payoff  for  the  using 
organization  [37:26]. 

Expert  systems  are  not  well  suited  for  problems  that 
require  the  senses  of  taste,  touch,  sight,  and  smell  [46:11- 
8].  A  simple  test  to  determine  if  the  task  or  problem  is 
dependent  upon  these  senses  for  unambiguous  communiction  is 
the  "telephone  test."  If  a  human  expert  can  help  the 
intended  user  of  the  expert  system  solve  the  problem  over 
the  telephone  then  the  problem  is  potentially  suited  for  an 
expert  system  [44:11-8]. 

The  problem  task  should  be  defined  very  clearly  in 
order  to  focus  the  development  of  the  expert  system  [33:27]. 
The  problem  should  be  limited  to  a  narrow,  specialized 
domain  in  order  to  keep  the  size  of  the  required  knowledge 
base  manageable  [3:66].  The  selected  task  should  be  neither 
too  easy  so  as  to  lack  credible  reasons  for  development  of 
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an  expert  system  nor  too  difficult  to  make  the  required 
assimilation  and  encoding  of  information  into  a  knowledge 
base  impractical  [37:28]. 

If  a  problem  cannot  be  solved  using  conventional  pro¬ 
gramming,  an  expert  system  may  offer  a  solution.  Expert 
systems  manipulate  knowledge  while  conventional  computer 
programs  manipulate  data  [51:24].  Expert  systems  can  offer 
solutions  to  problems  that  require  heuristic,  inferential 
processes  as  opposed  to  the  algorithmic,  repetitive  process¬ 
es  of  conventional  computer  programs  [51:24]. 

Advantages.  Expert  systems  offer  some  advantages  over 
human  experts  [2:3].  Humans  have  difficulty  dealing  with 
the  complexity  of  problems  in  today’s  work  environment. 
Even  when  complex  problems  are  broken  down  into  subcompo¬ 
nents,  being  able  to  understand  the  impact  of  a  decision  on 
other  areas  of  the  system  is  extremely  challenging.  An 
expert  system  lends  .support. to  the  decision-maker  in  complex 
problem  solving  activities.  Expert  systems  can  assimilate 
huge  amounts  of  data  and  examine  a  large  number  of  related 
factors  simultaneously.  Cost  reductions  are  achieved  as  the 
time  required  to  gather  and  process  data  is  reduced.  Con¬ 
stantly  changing  rules,  procedures,  and  methodology  are 
difficult  for  'experts'  to  assimilate.  Expert  systems  can 
quickly  assimilate  new  information  and  apply  the  information 
to  aid  management  in  the  decision-making  process  [47:197]. 


Expert  systems  are  not  biased  by  personality  emotion 
or  politics  [2:3].  Expert  systems  do  not  forget  key  fac¬ 
tors,  as  humans  occasionally  do  under  pressure.  Such 
systems  make  consistent  recommendations.  Such  a  system 
would  be  available  365  days  a  year,  24  hours  a  day.  "They 
don’t  get  sick,  go  on  leave,  travel  on  business,  transfer, 
or  retire  [2:3]."  Further,  the  real  experts  in  an  organiza¬ 
tion  are  very  busy  and  rarely  have  time  to  help  a  junior 
manager  work  through  a  time  consuming  problem. 

Expert  systems  can  automatically  keep  a  historical 
record  of  all  decisions  for  future  use  and  analysis.  The 
system  often  represents  the  first  written  record  of  the 
knowledge  in  the  domain  [3:20].  In  developing  an  expert 
system,  the  organization's  experts  must  closely  examine  the 
rules,  policies,  and  data  that  are  required  to  successfully 
accomplish  the  goals  of  the  organization.  This  intensive 
scrutiny  often  results  in  the  discovery  of  more  efficient 
and  effective  ways  of  doing  business  [2:4].  Once  the  system 
is  in  place  and  being  used,  the  human  expert  has  more  time 
to  devote  to  innovation  [2:4]. 

The  training  role  is  another  functional  area  where 
expert  systems  can  be  applied  advantageously.  New  personnel 
might  feel  uncomfortable  asking  the  boss  the  same  "stupid  " 
questions  over  and  over  again.  They  do  not  mind,  however, 
asking  a  "user  friendly  computer  tutor"  [2:4].  An  expert 
system  can  answer  the  question,  offer  a  solution,  and  also 
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explain  the  logic  behind  the  proposed  decision.  The 
system's  reasoning  is  easily  modified  if  a  flaw  is  detected. 
Expert  systems  are  easily  duplicated  and  copies  can  be 
distributed  wherever  the  expertise  is  needed.  Human  experts 
must  be  trained  and  educated  over  long  periods  of  time  [2:3- 


Figure  3  represents  experience  levels  in  a  typical 
organization.  The  group  with  the  most  experience  is  repre¬ 
sented  by  the  tallest  bars  on  the  far  left  [2:5].  In  most, 
cases,  these  experts  have  been  with  the  company  for  many 
years  and  are  nearing  retirement.  The  retirement  of 
personnel  with  so  much  accumulated  knowledge  represents  a 
serious  loss  of  expertise  to  the  organization.  Additional¬ 
ly,  a  shortage  of  experts  in  the  organization  curtails 
profitability  [23:197],  A  knowledge  engineer  can  change 
this  situation  by  interviewing  the  experienced  workers  and 
encoding  their  acquired  knowledge  in  an  expert  system. 

Under  ideal  conditions,  the  implementation  of  an  effective 
expert  system  allows  everyone,  including  the  new  employees, 
to  work  near  the  level  of  the  expert  as  depicted  in  Figure  4 
[2:5] . 

Disadvantages.  Expert  systems  also  have  certain  dis¬ 
advantages.  Expert  systems  are  not  applicable  in  every 
case.  If  forced  upon  an  organization  that  is  either  unpre¬ 
pared  or  unwilling  to  properly  apply  an  expert  system,  the 
potential  for  disruption  of  the  existing  organization  is 
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Figure  3 

Performance  of  a  Typical  Organization  [2:5] 
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Figure  4 

Level  of  Performance  with  an  Expert  System  [2:5] 
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considerable  [2:1].  Expert  systems  perform  best  for  limited 
task  situations.  A  complicated  management  task  might 
therefore  require  several  expert  systems  to  fully  address 
the  situation  [2:61. 

Expert  systems  are  very  difficult  to  construct  and  can 
require  a  great  deal  of  time  and  money  for  development  and 
implementation  [2:6].  Years  of  effort  by  high-priced  know¬ 
ledge  engineers  and  domain  experts  can  be  required  for  some 
expert  systems  [51:13].  Many  systems  require  specialized 
programming  and  expensive  computer  hardware  [47:202]. 

‘Extracting  expertise  from  an  expert  is  the  most  diffi¬ 
cult  aspect  of  creating  an  expert  system  [32:37].'  Expert 
system  development  involves  gleaning  information  from  human 
experts  who  often  do  not  know  in  explicit  detail  how  they 
solve  problems  and  tasks  [2:6].  Knowledge  engineering  can 
be  a  slow  and  inexact  process  [3:73]. 

Expert  systems  also  require  constant  updating  and  modi¬ 
fication  for  the  reasons  indicated  below: 

1.  Validation  efforts  frequently  reveal  faulty  or 
inconsistent  logic  which  must  be  corrected. 

2.  Systems  are  continually  expanded  to  handle 
additional  tasks. 

3.  Knowledge  in  the  field  evolves. 

4.  Policies  and  regulations  change. 

5.  Better  ways  of  performing  the  asx  are 
discovered  [2:6]. 

The  maintenance  and  upkeep  of  expert  systems,  especially  for 


systems  that  use  any  of  the  more  complex  programming 


languages,  requires  highly  trained  users  and  operators' 
Such  trained  personnel  are  currently  a  scarce  resource  in 
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most  organizations  [47:201-202]. 

A  major  potential  disadvantage,  described  by  Mike  Van 
Horn  in  his  book  Understanding  Expert  Systems  .  is  the 
"garbage  in  -  gospel  out"  syndrome  [47:6],  This  syndrome 
basically  describes  the  possibility  that  users  would  be 
afraid  to  override  the  system’s  recommendations.  In  my 
opinion,  too  great  a  dependency  on  any  decision-making 
technology  is  not  desirable. 

Current  rule-based  expert  systems  can  only  make  deci¬ 
sions  regarding  the  specific  rules  and  facts  entered  in  the 
knowledge  base  of  the  system.  Expert  systems  can  only  be 
used  for  the  narrow  task  that  the  system  was  designed  for 
[11:62].  Additionally,  the  required  scoping  of  expert 
system  problems  can  cause  myopic  judgements  to  be  offered  to 
the  user  of  the  system  ignoring  the  broader  problem  implica¬ 
tions  [27:35]. 

Testing  and  evaluation  of  an  expert  system  is  very 
difficult  [51:198].  A  developed  system  often  requires  an 
extensive  period  of  testing,  validation  and  refinement 
before  becoming  a  reliable  tool  [47:199].  The  problems 
addressed  by  expert  systems  do  not  generally  have  clear  cut 
right  or  wrong  answers  [3:84].  There  may  be  no  formal  way 
to  prove  the  answer  given  by  the  system  is  the  best  possible 
choice,  making  evaluation  of  the  system  difficult  [51:198]. 
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Trends .  In  solving  problems,  human  experts  rely  on  a 
combination  of  experience,  insight,  and  intuition  that  no 
knowledge  base  can  yet  match  [40:64].  Advances  in  natural 
language  programs  that  simplify  the  interaction  between  the 
user  and  the  computer  will  enhance  the  capabilities  of 
future  expert  systems.  Improved  graphics,  knowledge  acqui¬ 
sition  software,  and  information  networks  are  making  expert 
system  technology  an  increasingly  viable  management  tool 
[11:61].  The  number  of  people  trained  in  AI  ,  including 
knowledge  engineers  and  AI  language  programmers  is  rapidly 
rising  [47:203-204].  Eugene  Wang,  Vice  President  of  Gold 
Hill  Computers  states  in  a  1987  article  for  AI_  Expert : 

Over  two-thirds  of  Fortune  500  companies  currently 
have  AI  projects  staffed  and  under  way.  The  total 
AI  market  is  projected  to  grow  from  over  *1 
billion  in  1986  to  over  #4  billion  in  1990. 
Expert  systems  are  projected  to  grow  from  *145 
million  in  1986  to  *810  million  in  1990  [49:6]. 

Finally,  small  computers  are  gaining  more  power  and 
versatility.  Smal 1 . computers  have  greater  memory  capacity, 
and  are  less  expensive  than  in  the  past.  The  increased 
capabilities  of  small  computers,  particularly  microcompu¬ 
ters,  will  provide  potential  users  with  a  "low  risk 
opportunity  to  bring  expert  system  capabilities  in-house, 
and,  as  a  result,  determine  the  probable  value  of  this 
technology  for  their  applications  [31:207].’  ’Companies 
that  do  not  develop  expertise  in  tapping  this  technology 
within  the  next  few  years  will  be  at  a  competitive  disadvan¬ 
tage  [49:6].’  The  trends  described  above  will  facilitate 
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the  continued  growth  of  expert  systems  application  in  indus¬ 
try  and  will  increase  the  capacity  of  expert  systems  to 
enhance  productivity  [47:205]- 

The  background  information  presented  thus  far  was 
provided  so  the  reader  might  better  understand  the  research 
which  was  conducted.  The  basic  structure  and  function  of  an 
expert  system  was  discussed.  Expert  system  advantages, 
disadvantages,  and  trends  were  also  described. 

Statement  of  the  Problem 

Research  conducted  by  Dr.  Mary  K.  Allen  demonstrated 
that  an  expert  system  can  enhance  the  decision  quality  of 
inventory  managers  regarding  the  D041  system.  The  D041 
computes  replenishment  spares  requirements  for  reparable 
items  in  AFLC.  As  a  result  of  the  success  the  Inventory 
Manager  Assistant  (IMA)  expert  system  demonstrated,  Dr. 
Allen  suggested  continued  research  on  other  inventory 
management  tasks.  Her  research  included  a  listing  of  26 
inventory  management  tasks  that  were  considered  by  experts 
in  the  field  of  item  management  to  have  potential  for  expert 
system  application.  Asset  reconciliation  was  ranked  number 
6  out  of  26  potential  tasks  [3:112]  and  was  selected  for  the 
research  described  herein. 

Asset  reconciliation  is  the  process  of  comparing  world¬ 
wide  reported  assets  with  the  known  asset  position  and 
making  the  two  compatible.  An  asset  reconciliation  record, 
AFLC  Form  47,  is  used  for  the  computation.  A  sample  of  a 
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Form  47  is  depicted  in  Figure  5.  The  asset  reconciliation 


record  must  be  maintained  to  provide  an  accurate  record  of 
assets  in  the  system  [50:1],  Failure  to  properly  maintain 
these  records  can  cause  considerable  cost  increases  through 
the  improper  buy,  repair,  termination  and  disposal  of 
inventory  items.  Mission  readiness  is  reduced  when  needed 
items  are  not  procured  or  repaired  because  inaccurate 
records  do  not  reflect  the  need.  The  efficiency  of  inven¬ 
tory  management  operations  is  reduced  when  time-consuming 
research  is  required  to  correct  frequent  item  record 
inaccuracies . 

This  research  was  undertaken  to  validate  Dr.  Allen’s 
study  by  developing  an  expert  system  for  a  different 
inventory  management  task  and  measuring  the  resultant  change 
in  productivity  of  item  managers  who  were  assisted  by  the 
developed  expert  system. 

Research  Questions 

1.  Is  asset  reconciliation  a  suitable  problem  for 
application  of  an  expert  system? 

2.  Can  the  knowledge  needed  to  solve  the  selected 
problem  be  extracted  from  human  experts? 

3.  Can  the  rules  needed  to  perform  an  asset  reconcil¬ 
iation  be  encoded  using  the  Teknowledge  M.  1  software 
program’ 

4.  How  does  the  developed  expert  system  prototype 
affect  the  productivity  of  the  using  organization’ 
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Figure  5 

AFLC  Form  47  used  for  Asset  Reconciliation  [47:Atch  1] 
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Does  use  of  the  expert  system  increase 
effectiveness  by  at  least  15%? 


a . 

b.  Does  use  of  the  expert  system  reduce  the  time 
required  by  the  item  manager  to  complete  an 
asset  reconciliation? 

c.  What  is  the  user’s  subjective  evaluation  of 
the  assistance  provided  by  the  expert  system 
for  asset  reconciliation  computations? 

5.  How  significant  is  experience  level  in  determining 
performance  level  when  assisted  by  the  expert  system?  Does 
use  of  the  expert  system  help  less  experienced  item 
managers  more  than  experienced  item  managers? 

Scope  of  the  Study 

The  research  effort  was  limited  to  the  Warner  Robins 
Air  Logistics  Center  (WR-ALC),  Directorate  of  Materiel 
Management  (MM),  Robins  Air  Force  Base,  Georgia.  WR-ALC  was 
chosen  because  the  Director  of  Materiel  Management  expressed 
interest  in  the  project  and  promised  the  cooperation  and 
availability  of  needed  personnel  and  resources.  Research 
was  limited  to  the  Robins  AFB  site  due  to  time  constraints 
of  the  Graduate  Program  curricula  of  the  Air  Force  Institute 
of  Technology. 

The  selected  problem  task  was  bounded  to  include  only 
the  asset  reconciliation  computation  of  D041  items  using 
AFLC  Form  47.  Expert  systems  are  best  suited  to  narrowly 
scoped  problem  tasks  as  discussed  earlier  in  this  chapter. 
Limi tations 

The  expert  system  software  development  package  was 
preselected.  The  selected  package,  M.  1  by  Teknowledge,  is  a 


19 


Backward 


rule-based  system  that  uses  "backward  chaining, 
chaining  is  a  reasoning  process  that  works  backwards  from 

the  desired  solution  to  find  facts  that  support  the  solution 
[29:34].  Knowledge  base  rules  are  linked  or  "chained" 
together  [47:108].  The  backward  chaining  reasoning  process 
may  not  be  the  best  means  of  representing  inventory  manage¬ 
ment  knowledge;  therefore,  preselection  of  the  M.  1  knowledge 
acquisition  package  was  a  limitation. 

A  second  limitation  concerned  the  level  of  test  subject 
familiarity  with  computers  and  expert  systems.  Individuals 
selected  to  complete  an  asset  reconciliation  test  case  using 
the  expert  system  were  only  given  a  five  to  ten  minute 
introduction  explaining  how  to  use  an  expert  system  prior  to 
the  measurement  of  their  actual  test  performance  and  their 
time  to  completion. 

A  third  limitation  involved  the  restriction  upon  avail¬ 
able  Temporary  Duty  time  imposed  by  the  requirements  of  the 
Graduate  Program  curricula  of  the  Air  Force  Institute  of 
Technology.  The  depth  of  knowledge  engineering  and  perfor¬ 
mance  testing  was  reduced  by  this  constraint. 

Assumptions 

The  researcher  made  the  following  assumptions: 

1.  The  research  conducted  at  Sacramento  ALC 
listing  26  potential  inventory  management  tasks  for  expert 
system  application  is  applicable  to  Warner  Robins  ALC. 
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2.  The  results  of  the  present  research  regarding 
the  development  of  an  asset  reconciliation  expert  system  is 


applicable  to  all  the  Air  Logistics  Centers  despite  being 
conducted  solely  at  WR-ALC. 

Contr i buti ons  of  the  Research 

The  improved  management  of  inventory  resulting  from 
expert  systems  application  could  be  transformed  into  huge 
savings  [3:181].  AFLC  alone  has  responsibility  for  manage¬ 
ment  of  resources  valued  in  excess  of  50  billion  dollars 
[1:1].  The  estimated  total  U.S.  commercial  logistics 
expenditures  for  inventory  carrying  cost  for  1982  were 
listed  as  130  billion  [3:24].  These  figures  suggest  even  a 
small  percentage  improvement  in  resource  management  could 
yield  huge  savings. 

Development  of  an  expert  system  for  the  asset  reconcil¬ 
iation  task  appears  to  have  the  potential  to  alleviate 
critical  manpower  shortages  by  increasing  the  efficiency  of 
inventory  manager  activities.  Training  capabilities  will 
also  be  enhanced  by  the  development  of  an  expert  system  for 
asset  reconciliation. 

The  research  also  expanded  the  information  base  regard¬ 
ing  the  development  of  expert  systems  for  inventory 
management.  The  new  information  can  be  used  for  follow-on 
studies  thereby  enhancing  the  development  of  expert  system 
technology  in  the  Department  of  Defense. 
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Organization  o  f  the  Research  Report 

This  chapter  provided  background  information  on  expert 
systems,  stated  the  problem,  listed  research  questions  and 
discussed  the  scope  of  the  research.  Chapter  I  also 
summarized  the  limitations,  assumptions,  and  contributions 
of  the  research.  Chapter  II  reviews  the  relevant  literature 
regarding  expert  system  applications  and  the  measurement  of 
expert  system  performance.  Chapter  III  explains  the  metho¬ 
dology  followed  in  the  conduct  of  this  research.  Chapter  IV 
presents  the  findings  and  analyses.  Chapter  V  offers 
conclusions  and  recommendations. 
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1 1  .  Li terature  Review 

Overview 

The  purpose  of  this  chapter  is  to  review  the  literature 
in  order  to  provide  a  better  understanding  of  the  issues 
related  to  the  development  and  performance  measurement  of  an 
expert  system.  This  chapter  consists  of  three  sections. 
The  first  section  reviews  the  problem  selection  phase  of 
expert  system  development.  The  second  section  discusses 
expert  system  performance  evaluation  issues  and  validation 
criteria.  The  final  section  describes  actual  expert  system 
applications  and  their  associated  performance  in  terms  of 
efficiency,  effectiveness,  or  a  combination  of  both  effi¬ 
ciency  and  effectiveness. 

Problem  Selection 

This  section  will  first  review  problem  selection  issues 
and  then  discuss  the  problem  selection  methodology  used  for 
the  conduct  of  the  prior  research  being  validated. 

Problem  Selection  Issues.  Problem  identification  for 
the  expert  system  application  is  crucial  to  the  success  3  : 
the  developmental  effort  [3:55].  Initial  problem  assessment 
consists  of  the  following  activities: 

-  Identify  problem  domain  and  task 

-  Judge  appropriateness 

-  Consider  economic  justifications 

-  Identify  experts  and  the  nature  of  their  advice 


-  Identify  the  users  of  advice 

-  Identify  sample  cases  [3:56].' 

Figure  6  lists  criteria  for  potential  expert  system  tasks. 


1.  Choose  a  task  that  someone  performs  competently. 

2.  Good  domains  and  tasks  are  knowledge-intensive. 
Experience  has  sizeable  effect  on  the  solution. 

3.  Facts  are  acessible  and  generally  understood  by 
typical  users . 

4.  Facts  remain  stable  for  the  duration  of  a 
consul tation . 

5.  Solutions  can  be  enumerated. 

6.  Entities  in  the  domain  are  discrete. 

7.  One  single-instance  entity  is  clearly  at  the  heart 
of  the  task. 

8.  The  task  passes  the  'telephone  test. ' 


Figure  6 

Task  Selection  Criteria  [46:11-5  thru  11-9] 

Expert  system  problem  selection  criteria  can  be  sepa¬ 
rated  into  three  main  groups:  possible,  justified,  and 
appropriate  [51:127],  Figure  7  summarizes  the  problem 
domain  characteristics  that  make  expert  system  development 
possible.  Figure  8  summarizes  the  problem  domain  character¬ 
istics  that  make  development  of  an  expert  system  justified. 
Figure  9  illustrates  the  problem  domain  characteristics  that 
make  the  use  of  an  expert  system  appropriate. 
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Figure  7 

Necessary  Requirements  for  Expert  System  Development  [51:129] 


Nature 


Complexity 


Scope 


Figure  9 

Characteristics  that  Make  the  Use  of 
Expert  Systems  Appropriate  [51:132] 
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One  of  the  most  important  problem  selection  criteria  is  the 
existence  of  a  qualified  expert  [61:128],  Another  key 
criteria  according  to  Sandra  Cook,  vice-president  of  expert 
systems  at  MAD  Intelligent  Systems,  is  that  expert  systems 
'should  be  used  for  problems  that  can  be  satisfactorily 
solved  by  human  experts  at  such  a  high  level  that  somewhat 
inferior  performance  is  still  acceptable  [17:90].'  Problems 
should  require  satisfactory  versus  optimal  solutions  [3:67]. 

Problem  Selection  Me thodo 1 og y .  This  subsection  summa¬ 
rizes  the  methodology,  used  by  Dr.  Allen  in  1986,  to  select 
an  appropriate  problem  for  expert  system  development.  Dr. 
Allen's  objectives  during  the  problem  selection  phase  of  her 
research  entailed  the  completion  of  the  following  steps: 

"Step  1  -  Domain  Familiarization 

Step  2  -  Domain  Analysis 

Step  3  -  Expert  System  Workshop 

Step  4  -  Task  Selection 

Step  5  -  Expert  Selection  [3:98]." 

The  objective  of  Step  1  was  to  gain  familiarity  with  an 
item  manager's  tasks  and  basic  vocabulary.  Step  2  provided 
a  broader  understanding  of  the  item  manager’s  work  environ¬ 
ment.  During  Steps  3  and  4  the  expert  system  applications 
list  was  developed  from  which  the  asset  reconciliation  task 
was  chosen.  Step  5  involved  the  selection  of  experts  whose 
knowledge  wa3  encoded  in  the  expert  system  developed  [3:98- 


An  expert  system  workshop  conducted  at  Sacramento  ALC 
comprised  Step  3.  The  workshop  was  designed  to  prepare 
selected  item  managers  to  recommend  candidate  problems  for 
expert  system  application.  The  workshop  lasted  three  hours 
and  was  attended  by  a  total  of  ninety  item  managers.  The 
workshop  session  consisted  of  three  blocks  of  instruction. 
The  first  block  described  what  expert  systems  are,  advan¬ 
tages,  uses,  examples  of  existing  systems,  and  steps  in  the 
development  of  an  expert  system.  The  second  block  presented 
specific  selection  criteria  for  problems  well  suited  to 
expert  system  application.  The  third  block  was  used  as  a 
brainstorming  session.  Item  managers  were  asked  to  suggest 
candidate  expert  system  applications  [3:106-108]. 

Item  managers  were  also  encouraged  to  submit  candidate 
applications  on  three-by-five  index  cards.  Each  item  mana¬ 
ger  was  asked  to  submit  a  preferred  candidate  task  applica¬ 
tion  with  a  brief  description  of  the  advice  an  expert  system 
developed  for  this  task  would  provide.  Twenty-six  potential 
tasks  were  recommended  by  the  item  managers  [3:108].  A 
modified  Delphi  Technique  was  used  for  Step  4,  Task 
Selection.  The  Delphi  Technique 

'is  a  method  for  the  systematic  solicitation  and 
collation  of  judgements  on  a  particular  topic 
through  a  set  of  carefully  designed  sequential 
questionnaires  interspersed  with  summarized 
information  and  feedback  of  opinions  derived  from 
earlier  responses  [14:10].' 

Each  of  the  item  managers  who  attended  the  workshop  was 
given  a  Delphi  ballot  which  listed  each  of  the  twenty-six 
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I 

1 

1 


I 

I 

recommended  applications.  The  item  managers  were  asked  to 
select  and  rank  their  five  most  preferred  applications,  with 
number  one  representing  the  most  preferred  alternative 

i 

[3:  109]  . 

Eighty-six  item  managers  completed  the  Delphi  ballots. 

] 

,  The  rankings  were  weighted  as  follows: 

' Priority  Weight 


1 

2 

3 

4 

5 


5  points 
4  points 
3  points 
2  point 

1  point  [3:111] 


The  total  weighted  scores  for  the  eighty-six  item  managers 
are  listed  in  Table  1. 

Project  applications  receiving  total  weighted  scores 
greater  than  fifty  points  were  chosen  for  a  second  task 


selection  questionnaire.  Seventy-eight  item  managers  re¬ 
viewed  the  seven  projects  that  met  or  exceeded  the  fifty 
point  minimum  total  weighted  score.  The  item  managers  then 
rank  ordered  their  top  three  choices  using  the  following 
weighting  criteria: 


’  Priori  tv 


Weight 


1  3  points 

2  2  points 

3  1  point  [3:113]' 

The  total  weighted  scores  are  indicated  in  Table  2.  'Scrub¬ 
bing  the  D041  computation’  received  the  highest  ranking  and 
was  selected  as  the  task  for  which  Dr.  Allen  developed  the 
Inventory  Manager  Assistant  expert  system  in  1986  [3:113], 


30 


Table  1 

Total  Weighted  Project  Scores,  First  Questionnaire  [3:112] 


Project  To ta 1  We i ghted  Score 


1 . 

Excesses 

10 

2  . 

Allocation  Decisions 

13 

3  . 

Item  Analysis 

39 

4  . 

Controlled  Exceptions 

97 

5  . 

Backorder  Listing 

47 

6  . 

Repair  Requirements 

128 

7  . 

Cataloging  Action 

19 

8. 

MICAP  Support 

18 

9. 

Computer  Products 

48 

10. 

Initial  Provisioning 

35 

1  1  . 

PR’s 

31 

12. 

FMS  Support 

17 

13. 

S06's  (Budget) 

148 

14  . 

Critical  Item  Advisor 

44 

15  . 

Project  Focus 

8 

16. 

Shipment  Advisor 

13 

17. 

Asset  Reconciliation 

83 

18. 

Fill  Rate  Advisor 

15 

19  . 

Scrubbing  D041  Computation 

207 

20  . 

General  Training 

66 

21  . 

D041  File  Maintenance 

109 

22  . 

Special  Levels 

19 

23  . 

Amended  Shipping  Instructions 

5 

24  . 

PJJ/4 14 

23 

25  . 

D032/J041  Reconciliation 

36 

26. 

Other  Service  Support 

15 

Table  2 


:  Total  Weighted  Project  Scores,  Second  Questionnaire  [3:114] 

i 

i 

l 


Project 

General  Training 

Asset  Reconciliation 

Controlled  Exceptions 

D041  File  Maintenance 

Repair  Requirements 

S06's  (Budget) 

Scrubbing  the  D041 
Computation 


To tal  Weighted  Score 

36 

37 
40 
54 
94 
87 

126 


Asset  reconciliation  was  ranked  six  out  of  the  twenty-six 
potential  applications.  The  asset  reconciliation  task  was 
selected  for  expert  system  development  for  the  research 
described  in  the  present  thesis. 

Per f ormance  Evaluat ion 

Limited  information  on  the  evaluation  of  expert  is 
available  in  the  literature.  However,  considerable  litera¬ 
ture  exists  regarding  the  systematic  evaluation  of  informa¬ 
tion  systems.  The  relationship  between  information  system 
evaluation  and  expert  system  evaluation  was  considered  to  be 


worth  reviewing  along  with  literature  specifically  related 


to  expert  system  performance  measurement. 

Wh  e  n  to.  Evaluate.  Evaluation  should  be  informal 
through  the  early  stages  of  development  and  become  increas¬ 
ingly  formal  as  the  system  moves  toward  real-world  implemen¬ 
tation  [42:576].  Once  an  expert  system  prototype  is 
functioning  well  for  most  of  the  cases  presented,  a  more 
formal  evaluation  of  the  system  is  appropriate  [3:83]. 
Table  3  represents  a  chronological  progression  from  the  less 
formal  to  the  more  formal  stages  of  testing  [42:577]. 

During  Step  1  of  a  system’s  development,  explicit 
statements  which  define  the  measures  of  the  program’s 
success  and  the  methods  of  evaluating  failure  or  success  are 
required.  The  goal  of  Step  2  is  to  demonstrate  that  there 
is  an  appropriate  knowledge  representation  scheme  for  the 
task  domain.  The  appropriateness  of  the  knowledge 
representation  scheme  is  demonstrated  by  the  successful 
handling  of  a  few  test  cases  that  suggest  a  high-performance 
expert  system  is  possible.  Step  3  involves  informal  testing 
which  3eeks  to  define  major  problem  areas  in  the  system  and 
to  guide  the  iterative  process  of  system  development 
[42:577]  . 

Step  4  involves  the  assessment  of  the  program’s  actual 
utility  in  a  potential  user  environment  and  consists  of  a 
more  structured  evaluation  of  performance.  A  formally 
randomized  test  scenario  is  developed  to  show  the  system  is 
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Table  3 


*  -**  1 


'  Mmt 
«\  *’ 


Steps  in  the  Implementation  of  an  Expert  System  [42:577] 

1.  Top-level  design  with  definition  of  long-range 
goals 

2.  First  version  prototype,  showing  feasibility 

3.  System  refinement  in  which  informal  test  cases 
are  run  to  generate  feedback  from  the  expert 
and  users 

4.  Structured  evaluation  of  performance 

5.  Structured  evaluation  of  acceptability  to  users 

6.  Service  functioning  for  extended  period  in 
prototype  environment 

7.  Follow-up  studies  to  demonstrate  the  system's 
large-scale  usefulness 

8.  Program  changes  to  allow  wide  distribution  of 
the  system 

9.  General  release  and  distribution  with  firm  plans 
for  maintenance  and  upkeep 

capable  of  handling  almost  any  case  from  its  domain  of 
expertise.  A  successful  evaluation  at  Step  4  should  be 
attained  before  introducing  the  system  into  the  actual  user 
environment  [42:578]. 

Step  5  involves  system  evaluation  in  a  setting  where 
the  intended  users  will  have  access  to  the  system.  A  very 
limited  number  of  expert  systems  have  reached  this  stage  of 
assessment  [42:578].  Step  6  is  intended  to  gain  a  wider 


range  of  test  case  experience  and  to  develop  familiarity 
with  real  world  intricacies.  Step  6  is  commonly  referred  to 
as  "field  testing"  and  involves  extended  periods  of  on-site 
performance  evaluation  [42:578]. 

Step  7  involves  follow-up  studies  to  demonstrate  the 
system’s  large  scale  usefulness.  Formal  measurement  of 
pertinent  parameters  are  accomplished  before  and  after 
introduction  of  the  system  to  the  large  user  group.  The 
systems  efficiency,  cost  effectiveness,  and  the  systems 
acceptability  to  users  are  issues  of  concern  during  this 
step  of  developmental  evaluation  [42:578], 

Step  8  may  involve  programming  changes  or  modifications 
required  because  of  problems  discovered  during  the  previous 
step.  Once  the  problems  are  resolved,  wide-scale  distribu¬ 
tion  of  the  system  is  appropriate.  The  final  step.  Step  9. 
includes  the  on-going  evaluation  of  the  system  to  determine 
if  the  system  is  succeeding  in  the  broad  marketplace.  The 
system’s  credibility  is  likely  to  be  higher  if  the  first 
eight  steps  were  accomplished  giving  a  solid  base  of  data  to 
support  any  claims  of  performance  [42:5"T9]. 

Figure  10  represents  a  Management  Information  System 
(MIS)  evaluation  model.  Assessment  of  attitudes,  value 
perceptions,  information  usage,  and  decision  performance  are 
made  throughout  the  development  and  implementation  process 
of  the  MIS  [28:43]. 
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Assessments  of : 

Attitudes 
Value  Perceptions 
Information  Usage 
Decision  Performance 


Need  Assess  I 
MIS  Proposal 


Figure  10 

MIS  Evaluation  -  Conceptual  Model  [28:44] 


"It  is  good  programming  practice  to  test  any  program  in 
various  ways  throughout  its  development.  Testing  should  be 
integrated  into  the  deve 1 o p me n t - app 1 i c a t i on  cycle  [38:42]. 
Figure  11  depicts  an  example  of  a  staged  approach  for  the 
development  and  application  of  an  expert  system. 
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PHASE  I 


PHASE  II 


PHASE  III 


i 


PHASE  IV 


! 


PHASE  V 


PHASE  VI 

i 


i 

i 

[  Figure  11 

Staged  Approach  for  development  of  an  Expert  System  [20:451] 
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What  to  Evaluate-  An  important  step  in  any  evaluation 
is  to  determine  what  should  be  assessed.  Three  general 
areas  of  MIS  assessment,  attitudes  and  value  perceptions, 
information  usage,  and  decision  performance,  are  discussed 
be low. 

Atti tudes  and  Value  Perceptions.  Commonly  infor¬ 
mal,  pos t - i  mpl emen tat i on  assessment  of  a  MIS  is  all  that  is 
made.  Most  systems  are  developed  without  a  formal  evaluation 
of  the  attitudes  of  the  users  and  the  organization's 
managers  [28:45]. 

Decision  Pe  r  f  o  r  mane  e .  Decision  performance  re¬ 
flects  the  quality  of  the  decision-making  process  both  in 
terms  of  effectiveness  and  efficiency  [28:45].  The  evalua¬ 
tion  of  decision  performance  is  very  difficult  because  the 
nature  of  expert  systems  solutions  are  not  conducive  to 
clear  cut  right  or  wrong  answers  to  the  problems  addressed 
by  the  system  [3:83]. 

Information  system  evaluations  are  commonly  provided 
only  in  efficiency  oriented  terms  on  a  post  hoc  basis  by 
system  users  [28:45],  The  lack  of  ‘pre-vs-post"  system 
comparisons  is  a  serious  deficiency  in  the  assessment  of 
information  systems  [28:43].  The  emphasis  for  most  informa¬ 
tion  system  evaluation,  to  date,  has  been  efficiency 
oriented  rather  than  effectiveness  oriented  [28:43]. 

I n  formation  Usage ■  Related  to  decision  perform¬ 
ance  is  the  assessment  of  information  acquisition  and  usage. 
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A  new  system  may  affect  the  user’s  behavior.  Evaluation  of 


the  user's  procedures  in  obtaining  information  and  determi¬ 
nation  if  the  user  is  using  more  relevant  information  can  be 
measured  [28:45], 

A  related  evaluation  subject  is  the  measurement  of  the 
interaction  capabilities  of  the  system  with  the  user: 

1.  The  choice  of  words  used  in  the  questions  and 
responses  generated  by  the  program 

2.  The  ability  of  the  expert  system  to  explain 
the  basis  for  its  decisions  and  customize 
those  explanations  appropriately  for  the  level 
of  expertise  of  the  user 

3.  The  ability  of  the  system  to  assist  the  user 
when  he  or  she  is  confused  and  wants  help 

4.  The  ability  of  the  system  to  give  advice  and 
to  educate  the  user  in  a  congenial  fashion  so 
that  the  frequently  cited  psychological 
barriers  to  computer  use  are  avoided  [42:575] 

Performance  evaluation  should  measure  the  responsive¬ 
ness  of  the  information  system  [13:375].  The  responsiveness 
of  an  information  system  is  based  on  three  main  factors: 

"Power  -  the  degree  to  which  the  system  (including 
itshuman  element)  can  answer  the  most  important 
quest ions . 

Acessibility  -  the  degree  to  which  the  system  can 
provide  these  answers  in  a  timely  and  consistent 
manner . 

Flexibility  -  the  degree  to  which  the  system  can 
adapt  to  changing  needs  and  situations  [13:375]" 

Ho  w  t_o  Evaluate.  Due  to  the  nature  of  problems  that 
expert  systems  address,  evaluation  of  system  performance  is 
extremely  difficult. 
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Mo  one  knows  how  to  evaluate  human  expertise 
adequately  let  alone  how  to  evaluate  the  expert 
systems  that  attempt  to  recreate  that  expertise 
[22 : 277  ]  .  ' 

Further,  few  changes  affected  by  an  information  system 
can  be  so  isolated  that  their  effects  on  the  organization 
can  be  measured  directly  [13:614],  Methods  to  indirectly 
measure  the  effectiveness  of  the  user  as  a  result  of  inter¬ 
action  with  the  expert  system  exist.  These  methods  include: 

Significant  task  relevance  -  appox  i  mat,  ion  of  im¬ 
proved  decision  quality  through  users  subjective 
e  s  t  i  ma  t  e  s  . 

Willingness  to  pay  -  users  asked  to  specify  how 
much  they  are  willing  to  pay  for  a  specific  report 
or  system  capability. 

System  usage  -  appropriate  for  systems  whose  usage 
is  voluntary:  system  logs  may  permit  measures  of 
system  use  or  users  may  be  asked  to  estimate  their 
use  of  the  system. 

User  information  satisfaction  -  users  are  asked  to 
rate  their  satisfaction  with  such  aspects  of  the 
system  as  response  time,  turnaround  time,  vendor 
support,  accuracy,  timeliness,  format  of  outputs, 
and  confidence  in  the  system  [13:614]. 

Indirect  measurement  of  information  systems  are  most 
effective  when  taken  before  and  after  a  new  system  is 
introduced  [13:614].  This  allows  comparative  measurement  of 
operations.  System  measurement  is  not  optimal  shortly  after 
a  system  is  implemented  because  of  program  errors,  difficul¬ 
ties  in  learning  new  procedures,  and  general  resistance  to 
change  [13:614]. 

Evaluations  require  an  objective  standard  with  which 
the  results  of  the  evaluation  methodology  can  be  compared 


[42:579].  If  the  expert  system  can  suggest  solutions 
comparable  to  the  chosen  standard  for  the  selected  domain  , 
then  the  expert  system  should  be  more  convenient  and  less 
expensive  to  use  than  the  established  standard  [42:580], 

There  are  two  basic  ways  to  define  the  standard  used 
for  an  evaluation  of  an  expert  system: 

(1)  what  eventually  turns  out  to  be  the  correct 
answer  for  a  problem  (in  some  objective  sense)  or 

(2)  what  a  human  expert  (or  group  of  them),  pre¬ 
sented  with  the  same  information  available  to  the 
program,  say  is  the  correct  answer  [22:262], 

The  standard  used  to  evaluate  an  expert  system  must  be 
carefully  selected  as  this  standard  will  have  a  major  impact 
on  the  quality  and  the  results  of  the  study  [3:84],  ‘To 
date,  the  evaluation  of  expert  systems  has  primarily  been 
done  by  a  panel  of  experts  which  compared  the  expert  sys¬ 
tem’s  recommendation  to  the  recommendations  of  other  human 
experts  [2:11]."  When  using  the  decisions  of  the  domain 
experts  as  the  objective  standard  for  performance  evalua¬ 
tion,  the  domain  experts  themselves  should  be  rigorously 
evaluated  [42:580],  By  evaluating  the  experts  with  a  well 
accepted  and  documented  technique,  further  credibility  re¬ 
garding  the  evaluation  of  the  expert  system  is  achieved 
[33 : 580  ]  . 

In  determining  the  overall  validity  of  a  system, 
it  is  instructive  to  determine  how  well  human 
experts  do  in  the  problem  area  and  to  thereby 
create  reasonable  expectations  of  the  system’s 
performance.  If  human  experts  incur  a  15%  error 
rate,  should  the  system  be  expected  to  perform 
better7  If  so  how  much  better,  and  why  [33:46]7 


By  stating  the  methods  and  intent  of  an  evaluation 


experiment  potential  difficulties  can  be  avoided  and 
evaluation  results  are  less  open  to  misinterpret aion 
[22:278].  Table  4  offers  specific  guidance  to  be  considered 
during  the  evaluation  of  an  expert  system. 


Table  4 

Evaluation  Checklist  [22:278-279] 


1.  Specify  for  whom  the  evaluation  is  intended.  This 
greatly  influences  the  design  of  the  evaluation. 
Clearly,  the  informal  evaluations  usually  performed  for 
system  builders  and  expert  collaborators  would  not  be 
sufficient  for  financial  supporters  or  for  society. 

2.  Define  precisely  what  is  being  evaluated.  Isolate 
those  aspects  of  the  system  that  are  being  tested.  Is 
it  the  quality  of  the  system's  decisions  and  advice, 
the  correctness  of  the  reasoning  mechanism,  the  quality 
of  the  human-computer  interaction,  the  system’s  effi¬ 
ciency,  its  cost-effectiveness,  or  some  other  aspect'? 

3.  Select  an  appropriate  gold  standard  against  which 
to  compare  the  expert  system’s  performance. 

4.  Define  realistic  standards  of  performance:  it  may 
be  unreasonable,  for  example,  to  expect  better  perform¬ 
ance  from  the  expert  system  than  from  a  human  expert. 

5.  Specify  who  will  be  evaluating  the  results.  It 
could  be  human  experts  or  end  users  it  could  be 
individuals  or  a  group  evaluating  by  consensus ,  and  so 
on.  fNote  that  persons  evaluating  the  results  need  not 
be  the  same  as  those  for  whom  the  evaluation  is 
intended  .  ) 

6.  Eliminate  potential  bias,  for  example  by  avoiding 
presenting  the  results  in  the  form  of  computer  output, 
which  may  bias  judges  who  feel  negative  toward  computer 
technology . 

7.  Specify  the  type  of  test  cases  presented  --  random 
cases  preselected  to  range  over  a  broad  spectrum  of 
difficulty. 


Validation 


In  addition  to  the  processes  of  designing, 
developing,  and  implementing  expert  systems, 
validation  is  important  to  the  decision-making 
success  of  a  system  and  to  the  continued  use  of  an 
expert  system.  An  expert  system  that  has  not  been 
validated  sufficiently  may  make  poor  decisions. 
This  can  lead  to  a  loss  of  confidence  in  the 
particular  expert  system  or  in  other  systems, 
resulting  in  discontinued  use  and  financial  loss 
[35:468] . 

Validation  should  not  be  confused  with  verifica¬ 
tion,  Verification  involves  testing  of  the  accuracy  of  each 
rule  and  establishing  a  justification  for  each  rule  in  the 
knowledge  base  of  the  expert  system  [51:160].  Validity  is 
the  'extent  to  which  a  measure  accurately  represents  a 
variable  as  conceptualized  [16:42].'  During  system  valida¬ 
tion,  'the  knowledge  engineer  presents  cases  solved  by  the 
expert  and  the  prototype  system  to  other  experts.  This 
provides  a  way  to  compare  strategies  of  different  experts 
and  find  essential  points  of  disagreement  [51:160-161]."  The 
validation  process  requires: 

'1.  ascertaining  what  the  system  knows,  doea  not 
know,  or  knows  incorrectly 

2.  ascertaining  the  level  of  expertise  of  the 
system 

3.  determining  if  the  system  is  based  on  a 
theory  for  decision  making  in  the  particular 
d  o  ma i n 

4.  determining  the  reliability  of  the  system 

[35 : 469  ]  ' 

As  previous  analyses  have  noted,  expert  system  valida¬ 


tion  should  consist  of  periodic  informal  testing  rather  than 


a  single  formal  validation  at  the  end  of  the  project 
[35:469],  Ensuring  that  the  meaning  and  content  of  know¬ 
ledge  base  rules  meet  some  carefully  designed  criteria  of 
adequacy  is  the  basis  of  validation  [33:42],  'Defining  such 
criteria  is  the  key  to  successfully  conducting  a  validation 
procedure  and  demonstrating  the  level  of  acceptability  of 
the  knowledge  base  [33:42].' 

O’Leary  offers  a  framework  for  expert  system  validation 
procedures  outlined  in  Table  5.  The  first  step  in  the 
validation  process  involves  content  validity  [35:472],  Con¬ 
tent  validity  refers  to  the  determination  of  what  the  system 
knows,  does  not  know,  or  knows  incorrectly  and  can  be 
ascertained  by  direct  examination  of  the  system  components 
or  by  testing  the  system  [35:472], 

Criterion  validity  for  an  expert  system  validation,  the 
next  step  in  O’Leary's  validation  framework,  refers  to  the 
level  of  the  system's  expertise  [35:475].  A  comparison  of 
the  decisions  made  by  the  system  and  decisions  made  by  human 
experts  is  the  primary  method  used  for  expert  systems 
[35:475],  This  comparison  is  difficult  because  of  differing 
definitions  about  the  level  of  expertise,  knowledge-base 
criteria,  and  a  lack  of  clarity  about  what  is  being 
evaluated  [35:4751. 

The  next  step  in  the  framework  offered  by  O’Leary  is 
construct  validity  [35:473].  Construct  validity  tests  the 
basic  concepts  upon  which  the  expert  system  was  developed 


[35:476].  The  construct  validation  process  checks  for 
strong  relationships  where  they  are  supposed  to  exist  as 
well  as  checking  for  the  absence  of  relationships  where  they 
are  not  supposed  to  exist  [16:264].  Construct  validity 
suggests  that  an  empirical  approach  to  system  validation  is 
not  as  efficient  a  testing  process  as  an  approach  based  on 
the  basic  concepts  of  the  problem  domain  [35:476]. 

Objectivity  is  another  aspect  of  the  validation  pro¬ 
cess.  Objectivity  refers  to  minimizing  observer  variance, 
specifically  judgemental  variance  in  the  case  of  expert 
system  validation  [35:476].  Reduction  of  judgemental 
variance,  expert  bias,  can  be  achieved  by  administering  the 
test  independently  and  by  using  blinding  techniques 
[35:476].  Blinding  techniques,  for  example,  keep  the 
researcher  from  knowing  which  treatment  group  any  particular 
test  subject  belongs  to  [16:83]. 

The  cos t - bene f  i  ts  of  the  expert  system  should  also  be 
considered  [35:477].  System  benefit  is  difficult  to  measure 
and  is  ultimately  tied  to  how  the  system  will  be  used 
[35:477],  Two  factors  that  determine  the  cost  of  validating 
a  system  include  the  formality  and  the  depth  of  the  valida¬ 
tion  required  [35:477].  Cost-benefit  affects  the  scope  of 
validation  and  the  development  of  the  system  [35:477].  For 
example,  an  expert  system  that  requires  a  limited  input  of 
time  and  money  to  develop  and  implement,  generally  will  not 
receive  as  extensive  a  validation  as  an  expert  sytem  that 


will  require  a  substantial  commitment  of  resources  by  the 
using  organization. 

Reliability  refers  to  the  ability  of  the  system  to 
generate  ; dentical  solutions  given  identical  inputs 
[35:478].  "Reliability  refers  to  the  stability  or  consis¬ 
tency  of  the  values  that  are  obtained  [16:42]."  Expert 
system  reliability  testing  can  be  achieved  using  sensitivity 
analysis  or  measurement  against  standard  test  problems 
[35 : 478  ]  . 

"In  validating  an  expert  system’s  performance,  the  test 
problems  used  need  to  allow  the  validator  to  distinguish 
between  systematic  variance  and  chance  [35:478]."  Test 
problems  should  represent  the  type  of  problems  that  the 
human  expert  encounters  in  practice  [35:478],  A  range  of 
test  problems  from  simple  to  complex  should  be  used  in  the 
validation  process  [35:478]. 

Extraneous  variance  should  be  controlled  and  minimized 
[35:479].  The  influences  of  independent  variables  outside 
the  purposes  of  the  study  should  be  isolated  [35:479].  The 
complexity  of  the  expert  system,  location  in  the  system’s 
development  life  cycle,  amount  of  specialized  features  in 
the  system,  location  of  the  judges  during  testing,  and 
learning  by  the  judges  during  testing  can  influence  the 
quality  of  system  output  [35:479-480].  By  reducing  the 
number  of  variables  that  are  subject  to  change  the 
extraneous  variance  is  reduced  and  internal  validity  is 


Table  5 

Summary  of  Validation  Framework  [35:473] 


1.  Content  validity 

a.  Direct  examination  of  the  system  by  the  expert 

b.  System  test  against  human  experts  (Turing  test) 

(1)  Intraexpert  test 

(2)  Interexpert  test 

c.  System  test  against  other  models 

2.  Criterion  validity 

a.  Definition  of  the  level  of  expertise  of  the  system 

(1)  Human  evaluation  criteria 

(2)  Test  problems  to  define  the  level  of  expertise 

(3)  Quality  of  responses  defined 

b.  Knowledge-base  criteria 

c.  Clarification  of  evaluation  criteria 

3.  Construct  validity 

4 .  Obj  ectivity 

a.  Programmer  validation 

b.  Independent  administration  of  validation 

c.  Sponsor/end-user  validation 

d.  Biasing  and  blinding 

e.  Different  development  and  test  data 

5.  Economics  (cost-benefit) 

6 .  Reliability 

■  a.  System  test  against  itself  (sensitivity  analysis) 
b.  Test  problems  for  revalidation 

7.  Systematic  variance  (experimental  variance) 

a.  Problems  reflecting  range  of  problems  encountered 

b.  Variation  in  the  test  problems 

c.  Number  of  test  problems 

d.  Type  I  and  Type  II  errors 

8.  Extraneous  variance 

a.  Complexity  of  the  system 

b.  Expert  system’s  location  in  the  life  cycle 

c.  Recognition,  examination  and  testing  of  special  fixes 

d.  Location  of  judges  during  testing 

e.  Learning  on  part  of  judges 

9.  Error  variance 
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increased  [16:35]-  By  confounding  variables  the  number  of 
plausible  explanations  for  the  experimental  results 
increases . 

One  reason  why  researchers  prefer  to  do  experi- 
ments--to  manipulate  a  var i ab 1 e - - i s  that  manipula¬ 
tion  gives  them  the  opportunity  to  minimize  or 
avoid  confounding  of  relations  [16:37]. 

Error  variance  is  another  consideration  during  expert 
system  validation  [35:480],  Error  variance  can  be  caused  by 
a  variation  in  the  human  expert’s  responses  from  trial  to 
trial,  guessing,  momentary  inattention,  slight  temporary 
fatigue,  lapses  of  memory  and  transient  emotional  states 
[35:480  ].  This  form  of  variance  can  be  minimized  by  using 
proven  measurement  scales  and  by  asking  unambiguous  ques¬ 
tions  [35 : 480 ] . 

Marcot  considers  the  development  of  a  knowledge  base  to 
proceed  through  four  phases:  creating  the  prototype,  devel¬ 
oping  the  first  generation  rule  set,  testing  and  expanding 
the  rule  set  to  the  second  generation,  and  testing  the 
second  generation  rule  set  [33:42],  Specific  validation 
criteria  and  testing  procedures  should  be  integrated  within 
each  of  the  four  phases  [33:43], 

During  the  creation  of  the  prototype,  validation  helps 
define  the  original  problem  and  the  first  set  of  rules  to 
frame  the  inference  structure  [33:43],  Testing  of  the 
prototype  expert  system  should  determine  if  the  original 
problem  domain  was  narrowly  enough  defined  [33:43], 
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The  main  goals  of  validation  during  the  first  genera¬ 
tion  rule  set  phase  are  to  "reexamine  the  original 
objectives,  more  precisely  determine  the  problem  domain,  and 
establish  the  degree  of  detail  desired  in  the  system 
[33:44  ]."  This  validation  phase,  a  1 pha - t e s t i ng  stage, 
attempts  to  determine  the  accuracy  and  adequacy  of  the 
system  [33:44].  The  number  of  correct  predictions  made  by 
the  expert  system  are  compared  with  known  data  to  determine 
accuracy  [33:44],  Adequacy  may  be  expressed  as  a  simple 
fraction,  the  number  of  problem  domain  conditions  included 
in  the  expert  system  divided  by  the  total  number  of 
important  problem  domain  conditions  [33:44].  The  capacity 
of  the  knowledge  base  to  predict,  diagnose,  classify,  or 
monitor  within  a  specified  confidence  interval  is  also 
measured  during  the  alpha- testing  stage  [33:44]. 

During  the  expansion  to  a  second  generation  rule  set, 
the  characteristics  of  the  full-scale  system  should  be 
described  [33:44].  The  flexibility  or  adaptability  of  the 
system  for  future  applications  should  be  determined  during 
this  phase  of  validation  [33:44]. 

Testing  the  second  generation  rule  set  at  the  site 
where  the  system  will  be  used  involves  technical  and 
operational  validation  [33:44].  Testing  should  be  conducted 
to  measure  user  acceptance  of  the  system  and  to  determine 
how  well  tne  system  integrates  with  existing  procedural  and 
administrative  systems  in  the  problem  domain  [33:44  ]. 
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The  mam  steps  in  conducting  validation  tests  for  each 
phase  of  knowledge  base  development  include: 

'-defining  the  domain  and  context  within  which  a 
system  is  expected  to  perform  well  and  thus  the 
contexts  in  which  its  performance  is  poor  or 
unknown . 

-identifying  specific  performance  criteria  for 
val idation . 

-conducting  the  validation  tests  and  analyzing 
and  evaluating  the  results  [33:45].' 

Performance  criteria  should  be  specified  in  unequivocal 
terms  so  a  test  can  be  conducted  that  will  prove  or  disprove 
that  the  knowledge  engineering  has  accomplished  the  required 
task.  Formulation  of  specific  performance  criteria  also 
focuses  attention  on  the  precise  nature  of  the  initial 
conditions  and  the  final  output  that  the  system  is  expected 
to  produce  [23],  Specifiying  the  minimum  acceptable  per¬ 
formance  that  will  allow  the  system  to  be  considered  a 
success  and  involving  the  system  users  early  in  the  formula¬ 
tion  of  performance  criteria  is  important  to  the  success  of 
an  expert  system  [51:198].  Specific  criteria  that  may  be 
used  to  test  the  validity  of  a  knowledge  base  are  listed  in 
Table  6 . 

AppI ications  o f  Expert  Systems 

The  first  two  sections  of  this  chapter  have  reviewed 
the  process  of  selecting  a  problem  task  for  expert  system 
development  and  the  consequent  performance  measurement  of 
the  expert  system  application.  This  section  will  describe 
actual  expert  system  applications  and  their  reported 
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Table  6 


I  Validation  Criteria  [33:46-47] 

i 


I 

i 

Accuracy :  how  well  a  simulation  reflects  reality.  Compare 
inferences  made  by  rules  with  historic  (known)  data,  observe 
correctness  of  the  outcome. 

Adaptab 1 1 i tv :  possibilities  for  future  development  and 
appl ication . 

Adequacy :  the  fraction  of  pertinent  empirical  observations 
that  can  be  simulated. 

Appeal :  usability;  how  well  the  knowlege  base  matches  our 
intuition  and  stimulates  thought;  practicability. 

Avai labi 1 i tv :  existence  of  other,  simpler,  validated  knowl¬ 
edge  bases  that  solve  the  same  problem(s),  important  for 
determining  eventual  marketability. 

Breadth :  proportional  to  the  number  of  rules  used  in  the 
knowledge  base.  Determine  the  number  of  contexts  within 
which  the  system  should  be  expected  to  perform. 

Depth :  Determine  the  range  of  conditions  the  system  will 
address  and  which  parameters  are  necessary  to  diagnose, 
classify,  and/or  advise  for  each  condition. 

Face  val idi tv :  model  credibility.  Have  experts  review  sys¬ 
tem  throughout  development  project.  Compile  and  report 
results . 

General i tv :  capability  of  the  knowledge  base  to  be  used  with 
a  broad  range  of  similiar  problems. 

Precision:  capability  of  a  model  to  replicate  particular 
system  parameters;  number  of  significant  figures  used. 

Real  ism:  accounting  for  relevant  variables  and  relations. 
Establish  parameters  and  functions  in  the  rule  base  in  the 
same  terms  and  concepts  used  by  experts  or  end  user 
aud i ence . 

Re  liability :  the  fraction  of  model  predictions  that  are 
empirically  correct.  Ultimately  describes  statistical 
utility  of  the  likelihoods  in  the  rules  and  outputs. 


Table  6  (Continued) 


Resolution:  the  number  of  parameters  of  a  system  the  model 
attempts  to  mimic.  Identify  which  parameters  need  to  be 
defined  in  detail  and  which  do  not. 

Robustness :  conclusions  that  are  not  particularly  sensitive 
to  model  structure.  Determine  which  input  parameters  are 
least  and  most  significant  in  the  form  of  the  interim  and 
final  results  and  output. 

Sens  1 t i v i ty :  the  degree  to  which  variations  of  knowledge 
base  parameters  induce  outputs  that  match  historical  data- 
Specifically  determine  sensitivity  of  results  to  each  input 
parameter  by  varying  that  parameter  incrementally,  holding 
all  other  parameters  constant  and  matching  model  output  with 
historical  (known)  data. 

Technical  and  operational  val idi tv:  identification  and 
importance  of  all  divergence  in  model  assumptions  from 
perceived  reality.  Carefully  explicate  the  contexts,  condi¬ 
tions,  and  assumptions  that  underlie  the  rules. 

Turing  test:  assessing  the  validity  of  a  knowledge  base  by 
having  human  evaluators  distinguish  between  the  model’s 
conclusions  on  a  specific  problem  and  a  human  expert’s 
conclusions  solving  the  same  problem. 

Usefulness :  validates  that  the  system  contains  necessary  and 
adequate  parameters  and  relationships  for  use  in  problem¬ 
solving.  Usefulness  is  trivial  for  a  full-scale  system  but 
important  for  prototyping. 

ZilliliZ.:  a  knowledge  base’s  capability  of  producing 
empirically  correct  predictions.  Given  the  contexts  within 
which  the  system  is  expected  to  operate  well,  determine  how 
many  actual  conditions  the  sytem  can  accurately  diagnose. 

s.s:  the  number  of  processes  and  interactions 
reflected  in  the  model.  Consider  wholeness  in  light  of 
adap  tab i 1 i ty . 


performance  measurements  in  terms  of  efficiency,  effective¬ 
ness,  or  a  combination  of  both  efficiency  and  effectiveness. 
The  majority  of  literature  reviewed  described  expert  system 
performance  evaluation  in  terms  of  a  subjective  appraisal. 
Literature  offering  purely  subjective  appraisals  of  expert 
systems  were  not  included  in  this  review. 

Expert  systems  have  been  the  subject  of  considerable 
research  since  the  early  1970’s  but  have  only  recently  come 
into  widespread  use  for  business  applications.  Harmon  and 
King  state: 

Many  Fortune  500  corporations  are  assembling  AI 
departments,  venture  capitalists  are  rushing  to 
invest  in  entrepreneurial  expert  systems  compa¬ 
nies,  and  expert  systems  technology  is  well  on  its 
way  to  commercial  success  [23:1], 

Ef  fectiveness.  MYCIN  is  an  expert  system  designed  to 
assist  medical  specialists  in  the  diagnosis  of  infectious 
blood  diseases  and  to  prescribe  the  proper  antibiotics  for 
the  diagnosis  suggested  [47:6].  Five  faculty  fellows  in  the 
Stanford  Division  of  Infectious  Diseases  were  asked  to 
evaluate  15  cases  for  which  MYCIN  offered  therapy  advice. 
MYCIN  received  an  approval  rating  of  just  under  75%  of  the 
cases  tested.  A  second  study  of  MYCIN  performance  used  five 
evaluators  from  Stanford  and  five  from  medical  centers 
located  across  the  country.  Despite  the  refinements  made  to 
the  earlier  MYCIN  knowledge  base,  the  approval  rating 
remained  approximately  75%.  A  further  study  revealed  that 
even  infectious  disease  faculty  members  rarely  received  a 


70-80%  approval  rating  from  other  experts  in  the  field 
[42:582].  In  their  book  Artificial  Intelligence:  a  personal ■ 
co  mmons ens  e  j  ourney  .  William  R.  Arnold  and  John  S.  Bowie 
suggest  that  expert  systems  should  be  80%  reliable  and  that 
humans  have  been  about  60%  rel.  ible  in  their  decision  making 
historically  [7:68]. 

An  expert  system  developed  by  Ford  to  diagnose  mainte¬ 
nance  problems  with  robots,  covers  20%  of  the  rules 
regarding  robot  maintenance.  This  coverage  is  considered  by 
the  Ford  experts  to  account  for  60-80%  of  the  problems 
people  typically  encounter.  User  response,  after  initial 
resistance,  has  been  favorable  and  effectiveness  should 
increase  due  to  level  of  maintenance  problem  coverage 
provided  by  the  system  [12:24]. 

ASDEP  is  a  prototype  expert  system  for  designing  a 
power  plant’s  electrical  auxiliary  system  [26:56].  The 
expert  system  was  informally  evaluated  by  recognized  human 
experts  from  Southern  Company  Services  Incorporated  [26:62]. 
ASDEP  was  rated  against  four  factors:  load  groupings;  bus 
voltage  selections;  equipment  selections;  system  topology 
[26:62].  The  ASDEP  system  effectiveness  was  rated  favorably 
by  the  human  experts,  suggesting  power  plant  designs  simi¬ 
lar  to  those  suggested  by  the  human  experts.  The  economic 
efficiency  of  designs  was  not  considered  by  the  ASDEP  sys¬ 
tem.  Changes,  to  include  an  economic  evaluator  in  the 
system,  were  recommended  by  the  evaluation  team  [26:64]. 


Internist-1  is  an  expert  system  that  provides  internal 
medicine  diagnoses  [17:89],  The  system  misdiagnosed  18  out 
of  total  of  43  cases  [17:89],  Clinicians  at  Massachusetts 
General  Hospital  misdiagnosed  15  of  the  same  43  cases  tested 
with  the  expert  system  [17:89],  When  panels  of  doctors  were 
tested,  only  8  of  the  43  cases  were  misdiagnosed  [17:89], 

Helmut  Braun  of  the  University  of  Florida  and  John 
Chandler  of  the  University  of  Illinois  have  developed  an 
expert  system  that  predicts  intermediate  fluctuations  in  the 
movement  of  the  stock  market  for  nonconservative  investors 
[9:415],  The  system's  predictions  of  actual  market  move¬ 
ments  were  more  accurate  than  the  system’s  predictions  of 
the  expert’s  recommendations  [9:422],  The  system  correctly 
predicted  actual  stock  market  movements  for  64.4  percent  of 
the  example  cases  tested  [9:421],  In  comparison,  the  human 
expert  used  for  the  study,  correctly  predicted  stock  market 
movements  for  60.2  percent  of  the  108  example  cases  tested 
[9:421], 

Ef  f  lciencv.  A  marketing  expert  system,  More/2,  is 
designed  to  identify  direct  mail  buyers.  The  system  targets 
people  most  likely  to  respond  to  mail  order  saies.  The 
system  has  demonstrated  that  by  mailing  to  the  50%  of  an 
existing  list  most  likely  to  respond,  a  response  rate  equal 
to  70-80%  of  the  response  achieved  when  the  entire  list  is 


used  can  be  attained  [12:25]. 


SPINPRO  is  an  expert  system  that  makes  the  centrifuges 
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sold  oy  Beckman  Instruments  easier  to  use  and  more  efficient 
[49:5],  The  system  allows  lab  technicians  to  reduce 
centrifuge  run  times  by  as  much  as  70  percent  of  the  time 
formerly  required  [49:5]. 

PlanPower,  a  financial  planning  expert  system,  allows 
the  staff  of  Evensky  &  Brown  to  produce  a  moderately  complex 
financial  plan  that  once  required  approximately  50  hours  of 
computer  time  in  10  to  15  hours  [12:19]. 

Ken  Lindsay  and  Bob  Joy  of  Northrup  Aircraft  Division 
have  developed  an  expert  system  to  help  in  the  design  and 
manufacture  of  military  aircraft.  Their  system  has  reduced 
the  typical  process  planning  time  to  a  fraction  of  the 
original  8-12  hours  [12:22]. 

CORA  (Customer  Order  Relaying  Assistant)  is  an  expert 
system  used  by  Westinghouse  engineers  to  select  diverse 
relay  protection  system  configurations.  The  system  reduces 
the  time  required  to  configure  relay  systems  and  ensures 
that  the  relays  will  work  properly  [12:23], 

Major  Stephen  R.  L eclair,  USAF,  has  developed  a  multi- 
expert  knowlege  system  that  reduces  the  baking  time  for 
graphite  reinforced  lamination,  used  for  airframe  parts, 
from  12  hours  to  less  than  three  hours  [38:141]. 

Ef  fectiveness/Ef  f lciency.  The  Digital  Equipment  Corpo¬ 
ration  (DEC)  uses  a  combination  of  six  expert  systems  to 
automate  their  VAX  computer  sales,  engineering,  manufactur- 


ing,  and  customer  service  operations.  The  XSEL  expert  system 
designs  a  computer  system  to  meet  the  customer's  specific 
needs.  XCON,  in  operation  since  1979,  converts  the  design 
into  a  bill  of  materials  and  a  configuration  diagram.  XSITE 
generates  an  equipment  layout  from  the  configuration  diagram 
while  ISA  schedules  the  manufacturing  of  the  order.  ILOG 
helps  resolve  delays  during  the  manufacturing  process.  INET 
routes  the  order  from  the  factory  to  the  customer  [40:65]. 
The  series  of  expert  systems  just  described  can  now  design, 
package,  and  distribute  a  VAX  computer  system  better  than 
the  best  DEC  technician  [47:17].  The  system  is  saving  DEC 
20,000  dollars  a  month  [6]. 

Another  example  of  an  application  of  expert  systems  is 
the  Intelligent  Scheduling  and  Information  System  (ISIS) 
ieveloped  by  Carneg i e - Me  1 1  on  University.  ISIS  is  one  of 
several  systems  that  are  integrated  within  the  Intelligent 
Management  System  (IMS)  project.  IMS  is  a  subproject  of  the 
Factory  of  the  Future  project.  ISIS  is  now  used  by  Westing- 
house  to  schedule  the  efficient  use  of  its  own  job  shop 
activities  [27:38].  Most  of  the  costs  of  producing 
products  in  complex  organizations  can  be  attributed  to  over¬ 
head,  which  includes  managerial,  professional,  and  salaried 
personnel  costs  [19:32].  ISIS  is  a  step  toward  improving 
both  professional  and  managerial  productivity  by  reducing 
the  time  required  to  make  complex  management  decisions  and 
by  increasing  the  accuracy  of  the  decisions  made  [19:32]. 


Du  Pont  expects  to  create  2 (000  expert  systems  by  1991 
yielding  a  10  percent  increase  in  net  profits  this  equates 
to  approximately  *150  million.  Du  Pont  considers  payoff 
from  a  typical  successful  expert  system  to  be  10-to-l  or 
better.  A  new  system  could  save  the  company  as  much  as  a 
half  million  to  a  million  dollars  a  year  after  an  initial 
investment  of  only  *20,000  [52:36]. 

American  Express’s  authorization  assistant  is  an  expert 
system  designed  to  help  company  authorizers  decide  whether 
or  not  to  approve  individual  credit  card  charges.  Despite 
the  system’s  fledgling  status,  performance  to  date  suggests 
authorizer  productivity  will  improve  by  20-30X.  This  im¬ 
provement  translates  into  millions  of  dollars  for  American 
Express.  The  800  rule  system  has  impressed  its  human 
counterparts  with  the  quality  of  its  advice  [12:17], 

C0C0M01  is  a  microcomputer  based  expert  system  that 
allows  developers  to  estimate  project  cost  and  effort 
requirements  [53:52].  The  system,  developed  by  Barry  W. 
Boehm,  is  used  by  many  corporations  including  the  National 
Aeronautic  and  Space  Administration  [53:52].  The  system  can 
reduce  the  time  required  to  develop  an  estimate  from  half  a 
day  to  half  an  hour  [53:52].  Boehm  estimates  his  system 
yields  results  within  20  percent  of  actual  costs  68  percent 
of  the  time  [53:56].  An  act"al  corporate  user  of  the  system 
estimates  performance  of  the  expert  system  to  be  within  20 
percent  of  actual  costs  100  percent  of  the  time  [53:56]. 


Xerox  has  developed  an  expert  system  for  the  design  of 

paper  handling  systems  called  PRIDE  [34:102]-  The  PRIDE 
system  has  been  tested  on  actual  paper  transport  problems 
from  previous  and  current  copier  projects  [34:113].  PRIDE 
has  produced  satisfactory  designs  for  many  of  the  problem 
cases  tested  [34:113].  PRIDE  can  produce  a  design  that 
typically  takes  an  engineer  weeks  to  create  in  30  minutes 
[34:113].  PRIDE  also  performs  a  comprehensive  analysis  and 
report  generation  procedure  [34:113].  Xerox  is  still  in  the 
early  stages  of  system  evaluation  and  plans  to  report  more 
on  performance  in  later  papers  [34:113]. 

New  applications  of  expert  systems  in  the  DOD  and 
defense  related  industries  are  also  increasing.  For  example, 
the  Central  Integrated  Test  System  (CITS)  Expert  Parameter 
System  known  as  CEPS  is  a  leading  application  of  expert 
systems  to  aircraft  maintenance.  CEPS,  when  fully  imple¬ 
mented,  will  provide  enhanced  diagnostic  capabilities  to  B- 
1B  maintenance  technicians.  The  system  will  provide  a 
single  integrated  source  of  maintenance  history  and  list  on¬ 
going  diagnostic  activities.  CITS,  the  e  mbedded  diagnostic 
system,  will  provide  fault  detection,  fault  isolation,  and 
will  record  approximately  19,600  parameters  in  flight 
[36:211].  CEPS  will  reduce  fault  isolation  time  and  improve 
the  accuracy  of  diagnosis.  Initial  tests  demonstrated  a  30 
percent  reduction  in  'can-not-dupl icates '  and  'retest- 
okays',  and  a  30  to  60  percent  reduction  in  the  test  times 


for  line  replaceable  units  [5:3]  CEPS  is  anticipated  to 
save  over  *160  million  in  life  cycle  costs  [5:3]. 

Hughes  Artificial  Intelligence  Diagnostic  Expert  System 
(HAIDEX)  is  a  recent  research  effort  aimed  at  assisting 
depot  level  repair  activities  [5:2].  By  storing  past 
maintenance  history  of  problem  symptoms  and  the  eventual 
solution  HAIDEX  directs  troubleshooting  directly  to  the  most 
probable  fault.  HAIDEX  is  used  in  an  interactive  mode  with 
the  technician  during  the  troubleshooting  session.  Explana¬ 
tions  of  the  reasoning  used  by  the  system  are  invaluable  for 
training  and  debugging  the  system.  Evaluation  proved  the 
HAIDEX  prototype  to  be  equal  to  human  experts  in  solving 
difficult  problems  [5:3].  When  tested,  the  system  never 
made  an  erroneous  diagnosis  and  resolved  the  problems  in 
less  time  than  human  counterparts  who  were  tasked  to  solve 
the  same  problems.  HAIDEX  appears  to  be  superior  to  the 
novice  technician.  In  one  test  the  novice  worked  eight 
hours  on  a  problem,  then  gave  up,  HAIDEX  satisfactorily 
isolated  the  fault  in  36  minutes  [5:3]. 

The  Air  Force’s  first  expert  system  is  currently  being 
fielded.  IMA,  the  Inventory  Manager  Assistant,  was  devel¬ 
oped  by  Dr.  Mary  K.  Allen  to  assist  item  managers  with  the 
routine  data  review  of  the  following  elements  of  the  D041 
Requirements  Computation: 

Unit  Cost 

Date  of  Last  Procurement 
Administrative  Leadtime 
Production  Leadtime 


Base  Processing  Time 
Reparable  Intransit  Time 
Supply  to  Maintenance  Time 
Depot  Repair  Cycle  Time 
Turn-in  Time  [4:1] 

The  D041  Computation  System  computes  replenishment 
spares  requirements  for  reparable  items.  The  IMA  system  was 
developed  with  the  M.l.  shell,  a  product  of  Teknowledge,  and 
runs  on  any  IBM  compatible  microcomputer  with  512k  of  RAM. 
IMA  acts  as  a  consultant  to  item  managers  during  their 
quarterly  validation  of  requirements  for  reparable  items 
[3:4].  Dr.  Allen’s  research  showed  that  IMA  improved 
overall  item  manager  performance  on  both  simple  and  complex 
cases  between  7.6  and  10.4  percent  [3:185].  Performance  was 
improved  between  15.1  and  17.73  percent  for  complex  cases 
[3:185].  These  figures  strongly  suggest  that  expert  systems 
can  improve  the  decision  effectiveness  of  inventory  manag¬ 
ers.  Findings  regarding  the  ability  of  IMA  to  improve 
decision  efficiency  were  inconclusive  [3:185]. 

Summary 

The  objective  of  Chapter  II  was  to  review  the  litera¬ 
ture  in  order  to  gain  a  broader  understanding  of  the  issues 
involved  in  the  development  and  evaluation  of  an  expert 
system.  The  selection  of  a  suitable  problem  for  expert 
system  development  was  discussed  first,  followed  by  perform¬ 
ance  evaluation  issues  and  validation  criteria.  The  final 
section  of  the  chapter  reviewed  actual  expert  system 


appl ications . 


The  importance  of  selecting  an  appropriate  problem  for 
expert  system  development  was  discussed.  Task  selection 
criteria  were  reviewed  that  identify  problems  most  suited 
for  expert  system  application.  The  problem  selection  metho¬ 
dology  used  to  identify  asset  reconciliation  as  a  potential 
task  for  expert  system  development  was  then  described. 

The  literature  review  detailed  the  importance  of  an 
evaluation  procedure  that  is  integrated  throughout  the 
development  and  implementation,  of  an  expert  system.  Evalua¬ 
tion  should  include  assessments  of  attitudes,  information 
usage,  and  decision  performance.  The  difficulty  involved  in 
the  evaluation  of  an  expert  system  was  described  along  with 
techniques  to  resolve  the  problems  associated  with  perform¬ 
ance  measurement  of  expert  systems.  A  framework  for  the 
validation  process  of  an  expert  system  and  specific 
validation  criteria  were  presented  to  identify  to  the  reader 
issues  related  to  the  testing  of  an  expert  system. 

Expert  system  applications  were  reviewed  to  show  the 
range  of  uses  for  such  systems  and  to  describe  the  actual 
performance  me  as  ur  ements  of  the  systems.  The  majority  of 
literature  describing  expert  system  performance  was  purely 
subjective.  The  lack  of  objective  expert  system  performance 
measurement  was  attributed  to  the  difficulty  of  measurement. 

The  problem  selection  and  performance  evaluation  issues 
discussed  in  Chapter  II  were  incorporated  into  the 
methodology  presented  in  Chapter  III. 


62 


III.  Methodology 

Overview 

This  chapter  describes  how  the  following  research  ques¬ 
tions  were  addressed  and  tested. 

1.  Is  asset  reconciliation  a  suitable  problem  for 
application  of  an  expert  system9 

2.  Can  the  knowledge  needed  to  solve  the  selected 
problem  be  extracted  from  the  experts9 

3.  Can  the  rules  needed  to  perform  an  asset  reconcil¬ 
iation  be  encoded  using  the  Teknowledge  M.  1  software 
program9 

4.  How  does  the  developed  expert  system  prototype 
affect  the  productivity  of  the  using  organization9 

a.  Does  use  of  the  expert  system  increase 
effectiveness  by  at  least  15%9 

b.  Does  use  of  the  expert  system  reduce  the  time 
required  by  the  item  manager  to  complete  an 
asset' reconciliation9 

c.  What  is  the  user’s  subjective  evaluation  of 
the  assistance  provided  by  the  expert  system 
for  asset  reconciliation  co mputat l ons 9 

5.  How  significant  is  experience  level  in  determining 
performance  level  when  assisted  by  the  expert  system9  Does 
use  of  the  expert  system  help  less  experienced  item  managers 
more  than  experienced  item  managers9 

The  research  conducted  to  answer  each  of  these  ques¬ 


tions  was  composed  of  three  main  phases:  Problem  Selection; 


Knowledge  Engineering;  and  Performance  Measurement.  Each  of 
the  phases  are  discussed  in  turn  in  the  following  sections. 

The  overall  research  methodology  for  this  thesis  close¬ 
ly  followed  the  1986  research  design  used  by  Dr.  Allen 
during  her  development  of  the  Inventory  Manager  Assistant 
expert  system. 

Problem  Selection 

The  objective  of  this  phase  of  research  was  to  select  a 
problem  to  be  addressed  by  the  expert  system  and  to  choose 
domain  experts  whose  knowledge  would  be  encoded  into  the 
expert  system  to  solve  the  selected  problem.  This  initial 
phase  of  research  consisted  of  four  steps. 

Step  1_  2.  Task  Selection.  This  step  involved  a  review 
of  26  inventory  management  tasks  considered  to  be  most 
suited  for  expert  system  application  in  the  study  being 
validated  by  this  research.  The  methodology  used  to  develop 
this  listing  of  tasks  was  described  earlier  in  Chapter  II. 
The  researcher  considered  each  of  these  26  tasks  in  turn  as 
a  possible  task  for  expert  system  development.  Dr.  Allen's 
research,  listed  asset  reconciliation  among  the  seven  most 
promising  tasks  for  expert  system  development.  The  asset 
reconciliation  computation  was  also  considered  a  narrow 
enough  problem  task  for  the  thesis  project. 

Warner  Robins  Air  Logistics  Center  (WR-ALC)  ,  Director¬ 
ate  of  Materiel  Management  (MM),  Robins  Air  Force  Base, 
Georgia,  was  tentativley  chosen  as  the  location  for  the 
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project  since  the  organization  had  demonstrated  an  interest 
in  expert  system  development  in  the  past. 

Step  2_  Initial  Domain  Analysis-  Step  2  entailed  the 
researcher  reviewing  written  documentation  concerning  the 
asset  reconciliation  task.  The  objective  was  to  gain  a 
basic  familiarity  with  the  process  of  asset  reconciliation. 
Once  a  basic  knowledge  of  asset  reconciliation  was  attained, 
the  suitability  of  the  task  for  expert  system  development 
was  evaluated  using  the  following  selection  criteria: 

1.  There  are  available  experts  that  solve  the  problem. 

2.  Task  performance  is  significantly  effected  by 
knowledge  and  experience. 

3.  The  task  is  clearly  defined  and  easily  scoped. 

4.  The  task  involves  the  manipulation  of  large 
quantities  of  information  using  heuristics,  pattern 
matching,  and  search  procedures. 

5.  Solution  of  the  task  using  an  expert  system  is 
expected  to  provide  a  significant  payoff  for  the 
using  organization. 

6.  Strong  senior  management  support  exists  for  the 
development  of  an  expert  system. 

7.  Intended  users  of  the  system  would  accept  and  use 
the  resultant  system  [37:26-30]. 

The  researcher  considered  asset  reconciliation  to  meet  the 
necessary  requirements  of  the  first  five  criteria.  Discus¬ 
sions  with  Col  Gregory,  the  Materiel  Management  director,  at 
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Warner  Robins  Air  Logistics  Center  confirmed  senior  manage¬ 
ment  support  for  the  development  of  an  asset  reconciliation 
expert  system.  User  acceptance  was  considered  to  be  an 
unknown  factor. 

Step  3.  ^  Initial  Pro!  e  c  t  Mee  ting.  The  researcher  met 
with  key  personnel  at  WR-ALC/MM  to  describe  the  proposed 
effort.  Expected  project  deliverables  and  required  resources 
were  discussed. 

Expected  deliverables  included: 

1.  A  prototype  expert  system  for  asset  reconcil¬ 
iation. 

2.  Test  results  regarding  the  capabilities  as 
well  as  the  limitations  of  the  expert  system. 

3.  A  complete  documentation  of  the  expert  system. 

4.  A  demonstration  of  the  system  to  the  senior 
staff  and  to  item  managers  for  training. 

5.  A  follow-on  proposal. 

Expected  required  resources  included: 

1.  Human  expert’s  time;  four  hours  a  day  for  a 
minimum  of  two  weeks. 

2.  Thirty  asset  reconciliation  cases;  ten  for 
problem  identification,  ten  for  prototype 
development,  and  ten  for  evaluation  of  the 
prototype  expert  system. 

Problem  domain  information;  asset  reconcilia¬ 
tion  regulations,  training  manuals,  and  other 
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written  documentation  related  to  the  comple. 


tion  of  the  asset  reconciliation  task. 

4.  On-site  work  space  for  the  knowledge  engineer¬ 
ing  sessions,  knowledge  base  development,  and 
expert  system  testing  sessions. 

5.  Budget  requirements  were  to  be  handled  by  the 
Air  Force  Logistics  Command  and  the  Air  Force 
Institute  of  Technology. 

A  basic  calendar  of  events,  as  depicted  in  Figure  12, 
was  also  developed  during  this  step. 


Activity 

Date 

Domain  Familiarization 

1  Jan  -  1  Mar, 

1987 

M. 1  software  programming 
language  Familiarization 

1  Jan  -  1  Mar , 

1987 

Knowledge  Engineering 

Session  1 

23  Mar  -  27  Mar , 

1987 

Design  of  Initial 

Prototype  Expert  System 

30  Mar  -  12  Jun , 

1987 

Knowledge  Engineering 

Session  2 

15  Jun  -  19  Jun . 

1987 

Evaluation  of  Expert  System 
Performance 

22  Jun  -  26  Jun, 

1987 

Figure  12 

Calendar  of  Major  Events 
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Step  ±  z.  Expert  Selection.  Three  experts  were  chosen 
from  the  Materiel  Management  Requirements  Branch.  The 
selected  experts  had  attended  the  initial  project  meeting. 
Three  was  considered  the  optimal  number  based  on  several 
f  actors  : 

1.  Availability  of  personnel  was  constrained  due 
to  existing  on-site  job  requirements. 

2.  The  benefits  of  group  interaction,  multiple 

opinions,  and  a  range  of  experiences  was 
considered  by  the  researcher  to  be  important. 

3.  Optimum  number  of  experts  for  effective 
management  by  the  researcher  during  a 
knowledge  engineering  session. 

The  chosen  experts  were  recommended  by  the  senior  staff 
of  the  organization  as  having  extensive  knowledge  in  the 
area  of  asset  reconciliation.  The  sum  total  of  applicable 
experience  possessed  by  the  three  experts  equalled  92  years. 
The  communicative  skills  of  the  experts  were  also  consid¬ 
ered  in  the  selection  process.  The  ability  to  clearly  and 
accurately  express  the  actions  needed  during  the  asset 
reconciliation  process  was  of  primary  importance.  Comple¬ 
tion  of  this  step  concluded  the  first  phase  of  the  research. 
Knowledge  Engineering 

The  objective  of  this  phase  of  research  was  to  extract 
knowledge  from  the  selected  asset  reconciliation  experts  and 
create  a  knowledge  base  for  a  prototype  expert  system. 
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Step  1_  ^  Case  Collection.  Thirty  cases  of  actual  asset 
reconciliation  problems  computed  on  AFLC  Form  47  were 
collected  by  the  three  experts  chosen  during  Step  4  of  the 
previous  phase.  The  researcher  considered  thirty  cases  to 
be  a  sufficient  number  of  cases  to  offer  a  range  of 
situations  yet  still  be  a  manageable  number  to  work  with. 
Cases  were  selected  by  the  inventory  management  experts  to 
represent  trivial,  typical,  and  extreme  cases  of  the  asset 
reconciliation  task.  The  researcher  then  systematically 
divided  the  cases  into  design  cases,  training  cases,  and 
evaluation  cases,  as  suggested  by  Teknowledge  Incorporated 
[40:11-19].  Design  cases  represented  a  small  set  of  typical 
problems  that  the  initial  expert  system  prototype  attempted 
to  solve.  The  training  cases  were  a  larger  set  of  cases 
used  for  expansion  of  the  knowledge  base.  The  evaluation 
cases  were  used  to  demonstrate  the  capabilities  and  limita¬ 
tions  of  the  developed  expert  system  [46:11-19], 

Step  2  z.  Knowledge  Engineering  Sessions .  The  decision 
rules  used  by  the  experts  to  solve  asset  reconciliation 
tasks  were  obtained  during  this  step.  The  Nominal  Group 
Technique  (NGT)  along  with  the  case  analysis  method  was  used 
to  gain  this  information.  NGT  is  a  structured  group  meeting 
that  includes  the  following  steps: 

1)  Silent  generation  of  ideas  in  writing. 

2)  Round-robin  feedback  from  group  members 
to  record  each  idea  in  a  terse  phrase 
on  a  flip  char t . 


3)  Discussion  of  each  recorded  idea  for 
clarification  and  evaluation. 

4)  Individual  voting  on  priority  ideas  with 
the  group  decision  being  mathematically 
derived  through  rank  ordering  or  ratings 
[14:8]. 

The  three  experts  met  and  worked  as  a  group  at  all  of 
the  knowledge  engineering  sessions.  Two  distinct  series  of 
meetings  were  held  with  the  experts.  The  first  series 
consisted  of  four  hour  sessions  each  day  for  five  days.  A 
fourth  expert  was  brought  into  the  knowledge  engineering 
process  on  the  third  day  of  the  first  series.  The  fourth 
expert  was  much  less  experienced  than  the  other  three 
experts.  This  fourth  expert  had  only  been  an  item  manager 
for  seven  years.  He  was  actively  performing  asset  reconcil¬ 
iation  as  the  primary  function  of  his  daily  activities.  The 


fourth  expert  was  recommended  by  his  supervisor  as  having  a 
good  understanding  of  the  asset  reconciliation  process  and  a 
strong  interest  in  the  computer  automation  of  the  asset 
reconciliation  Form  47.  The  fourth  expert  was  added  to  the 
knowledge  engineering  process  in  an  attempt  to  elicit  more 
of  the  fundamental  reasoning  processes  involved  in  the  asset 
reconciliation  computation.  During  the  second  series  of 
meetings  the  knowledge  engineer  met  with  the  four  experts 
four  hours  a  day  for  five  days.  All  the  group  sessions  were 
tape  recorded. 

Day  One  of  the  first  series  of  meetings  was  spent 
developing  a  basic  understanding  of  the  problem  task.  A 


XX ' 


brief  review  of  the  initial  project  meeting  was  accomplished 
first.  Exercise  One  was  then  completed  using  the  NGT. 
During  Exercise  One  the  experts  were  asked  to  answer  five 
statements  regarding  the  asset  reconciliation  process: 

1.  Define  Asset  Reconciliation  in  general  terms. 

2.  What  do  you  consider  the  most  difficult  part  of 
the  Asset  Reconciliation  process? 

3.  What  is  the  most  common  mistake  made  during  the 
Asset  Reconciliation  process? 

4.  Do  certain  assets  require  significantly  different 
procedures  during  the  Asset  Reconci lai t i on  process? 
If  so,  what  types  of  assets  and  how  are  the 
procedures  different? 

5.  What  is  the  objective  of  completing  AFLC  Form  47? 

A  copy  of  the  written  directions  given  to  the  experts  during 
Exercise  One  is  included  in  Appendix  B.  A  concensus  regard¬ 
ing  the  actual  scope  of  the  task  was  achieved  using  Exercise 
One  and  the  NGT.  Following  Exercise  One,  the  experts  were 
asked  to  select  two  ‘typical*  cases  from  the  ten  asset 
reconciliation  cases  the  researcher  provided  for  them.  The 
cases  distributed  to  the  experts  were  from  the  original 
thirty  cases  collected  during  Step  1.  The  three  expert’s 
each  individually  selected  two  typical  cases,  for  a  total  of 
six  cases  selected  by  the  experts.  Then,  as  a  group,  the 
experts  chose  three  cases  from  the  six  typical  cases.  These 
three  cases  were  used  during  Exercise  Two.  Each  expert  used 


one  of  the  typical  cases  selected  to  conduct  an  asset 
reconciliation  computation.  The  experts  were  asked  to  write 
out  in  specific  detail  how  they  would  complete  the  computa¬ 
tion  and  where  they  would  find  the  information  required  for 
the  computation.  A  copy  of  the  written  directions  given  to 
the  experts  for  Exercise  Two  is  located  in  Appendix  C. 

Day  two  of  the  first  knowledge  engineering  session 
began  with  a  review  of  the  expert’s  responses  to  Exercise 
One  and  Exercise  Two.  Each  of  the  expert’s  written 
descriptions,  from  Exercise  Two  detailing  how  to  complete  an 
asset  reconciliation  computation,  were  examined  and  com¬ 
mented  on  by  the  group. 

Exercise  Three  was  conducted  on  the  third  day  of  the 
knowledge  engineering  session.  Exercise  Three  was  used  to 
elicit  a  more  detailed  analysis  of  the  questions  that  need 
to  be  asked  during  an  asset  reconciliation  computation.  The 
approach  was  modeled  after  an  experiment  conducted  by 
Coopers  &  Lybrand  during  their  development  of  the  ExperTAX 
expert  system  [30:23].  Their  experiment  was  video  recorded 
and  used  three  domain  experts,  a  novice,  and  a  curtain.  The 
objective  of  the  experiment  was  to  mimic  a  computer  by 
having  the  three  experts  act  as  the  expert  system,  the 
curtain  act  as  the  computer  screen  and  the  novice  act  as  the 
user  [30:23]. 

The  experts  participating  in  Exercise  Three  were  asked 
to  imagine  a  computer  screen  separating  them  from  each 
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other.  Expert  One  was  seated  at  a  table  with  all  the 
supporting  documentation  to  perform  an  asset  reconciliation. 
Expert  two  was  seated  across  the  table  from  Expert  One  and 
directed  to  lead  Expert  One  through  the  asset  reconciliation 
process.  Expert  Two  was  not  allowed  to  use  hand  motions  and 
was  asked  to  offer  directions  to  Expert  One  in  the  form  of 
questions  and  advice.  Expert  Two  was  asked  to  respond  to 
Expert  One’s  answers  in  the  same  way  as  an  expert  system 
computer  program  might  respond,  by  offering  solutions  and 
prompting  further  actions.  A  copy  of  the  written  directions 
provided  to  the  experts  for  Exercise  Three  is  located  in 
Appendix  D. 

The  remaining  time  during  the  first  series  of  meetings 
was  spent  discussing  the  elements  of  AFLC  Form  47  computa¬ 
tions  and  the  sequence  of  events  that  occur  during  a  typical 
asset  reconciliation  computation.  Subcategories  of  the 
problem  task  were  identified  and  prioritized  according  to 
importance  and  the  frequency  of  being  incorrectly  computed. 
Cases  were  reviewed  in  detail  by  the  experts  and  the 
knowledge  engineer.  The  experts  were  asked  to  silently 
generate  their  solutions  and  decision  protocol  for  the 
elements  of  the  computation  that  the  expert  system  was 
addressing.  Their  solutions  were  then  examined  by  the  group 
and  a  consensus  achieved  through  open  discussion. 

On  the  final  day  of  the  first  knowledge  engineering 
session,  a  simple  introductory  expert  system  for  asset 
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reconciliation  was  developed  consisting  of  six  rules  using 
the  M.  1  software  package.  The  introductory  system  was 
demonstrated  to  the  experts  to  provide  a  better  understand¬ 
ing  of  how  an  expert  system  would  function  and  to  stimulate 
ideas  regarding  the  further  development  of  the  asset 
reconciliation  expert  system. 

Step  3.  Building  the  Initial  Prototype  System.  Tape 
recordings  of  the  first  knowledge  engineering  sessions  were 
carefully  reviewed.  The  original  knowledge  base  was  encoded 
based  on  the  facts  and  rules  obtained  from  the  experts 
during  the  first  series  of  meetings.  One-hundred  specific 
rules  regarding  the  asset  reconciliation  computation  were 
included  in  the  original  knowledge  base.  The  M.  1  shell  was 
used  to  develop  the  first  knowledge  base  system.  M.l  was 
selected  in  order  to  replicate,  as  closely  as  possible,  the 
study  being  validated.  Completion  of  this  step  resulted  in 
the  development  of  the  initial  expert  system  prototype 
called  the  Manager's  Asset  Reconciliation  System  (MARS). 

Step  4.  2.  Expanding  and  Verifying  the  System.  During 
the  second  series  of  meetings  with  the  experts,  the  MARS 
prototype  underwent  an  initial  verification.  The  knowledge 
engineer,  i.e.  this  researcher,  again  met  with  thethree 
original  experts  for  five  days,  four  hour  sessions  each  day. 
After  each  of  the  daily  four  hour  sessions,  the  fourth 
expert  was  consulted  on  an  individual  basis  regarding  the 
issues  discussed  during  the  sessions  with  the  three  experts. 
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The  individual  consultations  with  the  fourth  expert  provided 
a  greater  variety  of  explanations  regarding  asset  reconcil¬ 
iation  activities.  A  Zenith  248  micro-computer  was  used 
throughout  the  verification  process  to  add  and  update  rules 
in  the  knowledge  base  of  the  expert  system. 

The  experts  were  provided  with  a  listing  of  the 
knowledge  base  coded  in  the  M.  1  language  at  the  beginning  of 
the  first  session.  The  researcher  then  explained  the  syntax 
and  the  logic  that  M.  1  used  to  reach  conclusions.  The 
experts  were  then  asked  to  review  the  knowledge  base  listing 
three  separate  times.  During  the  first  review,  the  experts 
checked  for  any  incorrect  rules.  The  second  review  entailed 
a  check  for  faulty  wording  of  the  questions  or  advice  listed 
in  the  M.  1  coding.  The  experts  were  asked  to  provide  addi¬ 
tions  or  recommended  changes  in  terminology  which  would  make 
the  questions  and  advice  clearer  to  a  user  of  the  system. 
During  the  third  review  the  experts  were  asked  to  check  the 
order  in  which  questions  were  asked  and  advice  given.  The 
experts  were  asked  to  recommend  changes  that  would  make  an 
interactive  consultation  with  the  system  a  more  logical 
progression . 

Following  the  verification  of  the  knowledge  base  list¬ 
ing,  the  experts  were  introduced  to  the  MARS  prototype.  The 
experts  each  ran  several  interactive  consultations  with 
MARS.  The  researcher  used  the  remainder  of  the  first  day  to 
make  changes  to  MARS  recommended  by  the  experts.  This 
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correction  procedure  was  accomplished  each  day  such  that  the 


system  was  ready  for  further  verification  the  following  day. 

The  second  day  began  with  another  review  of  the  M.  1 
syntax  with  the  experts.  The  rest  of  that  day  and  the 
remainder  of  the  week  were  spent  working  through  ten  design 
cases  and  five  training  cases  with  the  experts  using  MARS. 
The  experts  each  took  turns  working  at  the  computer 
terminal,  keying  the  inputs,  and  reading  advice  offered  by 
MARS  to  the  other  experts.  The  other  two  experts  worked 
manually  through  the  cases  and  would  agree  or  disagree  with 
MARS’  advice  based  upon  their  manual  computations.  When  a 
disagreement  arose  MARS  was  requested  to  show  which  rule  had 
been  invoked  to  provide  the  advice.  The  rule  was  then 
checked  for  accuracy. 

After  verification  with  the  three  experts  was  concluded 
on  the  morning  of  the  fifth  day,  an  experienced  item 
manager  and  a  novice  item  manager  were  introduced  to  MARS 
and  asked  to  critique  the  system.  These  item  managers  were 
selected  because  they  were  less  experienced  than  previous 
users  of  MARS.  Neither  the  experienced  item  manager  nor 
the  novice  item  manager  had  participated  in  any  earlier 
activities  regarding  the  development  of  MARS.  They  offered 
an  independent  evaluation  of  the  system.  One  of  the  experts 
spent  three  hours  working  through  two  training  cases.  The 
other  experts  were  also  present  to  explain  MARS’  reasoning 
if  the  experienced  or  the  novice  item  manager  disagreed  with 


MARS’  output.  Several  changes  resulting  in  the  simplifica¬ 
tion  of  the  system’s  terminology  were  recommended  as  a 
result  of  this  session. 

Upon  completion  of  the  verification  sessions  both  the 
original  three  experts  and  the  fourth  expert  agreed  that  the 
demonstrated  performance  of  MARS  warranted  formal  testing. 
Per f ormance  Measurement 

The  purpose  of  this  phase  of  research  was  to  determine 
if  the  prototype  expert  system  enhanced  the  productivity  of 
inventory  managers  using  the  system.  A  sample  consultation 
with  MARS  and  the  resultant  output  are  included  in 
Appendices  I  and  J  respectively.  The  following  research 
questions  were  answered: 

How  does  the  developed  expert  system  prototype 

affect  the  productivity  of  the  using  organization? 

Does  use  of  the  expert  system  increase 
effectiveness  by  at  least  15%"? 

Does  use  of  the  expert  system  reduce  the  time 
required  by  the  item  manager  to  complete  an 
asset  reconciliation"? 

What  is  the  user’s  subjective  evaluation  of 
the  assistance  provided  by  the  expert  system 
for  asset  reconciliation  computations'? 

How  significant  is  experience  level  in  determining 
performance  level  when  assisted  by  the  expert  system"?  Does 
use  of  the  expert  system  help  less  experienced  item  managers 
more  than  experienced  item  managers'? 

Step  X  2.  Experimental  Model  Development.  A  single 
experiment  was  used  to  measure  decision  effectiveness  with  a 
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two-by-two-by-three  between  subjects  factorial  design- 
subject  participated  in  only  one  of  the  cells  in  the  design. 
Figure  4  depicts  the  cell  design  for  this  experiment.  The 
levels  and  treatments  of  the  experiment  are: 


Treatment 


Levels 


1 .  Task  Method 

1  . 

With  Expert  System 

2  . 

Manual ly 

2.  Task  Difficulty 

1  . 

S i mp 1 e 

2  . 

Complex 

3.  User  Experience 

1  . 

Novice 

2  . 

Journeyman 

3. 

Advanced 

The  experimental  model  for  the 

effectiveness/efficiency 

testing  can  be  depicted  as 

f ol lows 

: 

x  =a+y3+y  +  afi 

+  or  y 

+  0Y  +  ct0Y  +6 

ijkl  i  j  k  ij 

ik 

jk  ijk  ijkl 

Where  i 

=  2 

1 evels 

o  f 

task 

method 

j 

=  2 

levels 

of 

task 

di f  f icul ty 

k 

=  3 

levels 

o  f 

user 

experience 

1 

=  cell  replicate 

The  following  definitions  were  used  for  development  of 
the  experimental  model: 

Novice  -  up  to  one  year  of  experience  as  an  item 
manager 

Journeyman  -  one  to  four  years  experience  as  an  item 
manager 


Advanced 


more  than  four  years  of  experience  as  an 
item  manager  (senior  inventory  managers 
considered  four  years  to  be  the  learning 
curve  level-off  point  for  most  item 
managers)  [3:332-333] 


Ef  f iciency 


the  time  to 


complete 


the  selected  task. 
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Task  Method 


Manual 


Type  of  Case  Subject  Experience 

Novice 

Simple  -  Journeyman 

Advanced 


Complex 


Novice 

Journeyman 

Advanced 


Novice 

Journeyman 

Advanced 


Figure  13 

Cells  and  Cell  Sizes  of  the 
Effectiveness/Efficiency  Experiment 


Cell 

Size 

6 

9 

10 

7 

8 
5 


6 

10 

9 


8 

8 

8 


Effectiveness  -  performance  score  between  0  and  100 

points  related  to  the  accuracy  of  test 
subject's  answers  as  compared  to  a 
test  key  developed  for  the  test  cases 

[3:329] 
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Task  performance,  for  both  simple  and  complex  test 
cases,  was  scored  using  the  "on-hand  assets",  block  15, 
quantity  from  the  Form  47.  Four  other  parameters  listed  on 
the  Form  47  were  also  informally  checked  to  verify 
procedural  accuracy. 

1.  Total  acquired  assets 


2.  Starting  position 

3.  Balance  accountable  assets 


4.  Net  balance  accountable 


The  test  subjects  determined,  either  manually  or  with 
the  assistance  of  MARS,  a  quantitative  value  for  the  "on- 
hand  assets"  and  the  four  parameters  listed  above  during  an 
asset  reconciliation  computation.  The  test  subject’s 
answers  were  compared  to  the  correct  answers  for  the  same 
problem  as  determined  by  the  MARS  developmental  experts  and 
their  immediate  supervisor.  Deviation  of  the  test  subject's 
answers  from  the  correct  answers  was  measured  according  to 
the  following  predetermined  descriptive  categories:  very 
good  answer,  good  answer,  mediocre  answer,  poor  answer,  very 
poor  answer.  Answers  falling  in  each  of  the  categories  were 
assigned  the  following  respective  point  values:  100,  75,  50, 
25,  and  0.  The  subject  could  score  between  0  and  100  points 
on  the  effectiveness  portion  of  task  performance  testing. 
Efficiency  was  measured  as  the  time  to  complete  the  asset 
reconciliation  test  case.  The  test  subjects  were  not  aware 
that  they  were  being  timed. 
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Step  2  2.  Test  Case  Selection.  This  step  involved  the 
selection  of  a  simple  and  complex  case  to  be  used  for 
testing  of  the  expert  system.  "A  complex  case  was  defined 
as  a  very  difficult  case  which  would  be  very  infrequently 
encountered.  A  simple  case  was  defined  as  a  very  easy  case 
which  would  be  very  frequently  encountered  [3:130]."  A 
sample  of  a  complex  case  is  included  in  Appendix  H. 

Three  of  the  four  developmental  experts  were  directed 
to  select  a  simple  case  and  a  complex  case  from  their  asset 
reconciliation  files.  Each  of  the  three  experts  were  then 
asked  to  grade  their  own  cases  and  the  cases  selected  by  the 
other  two  experts  according  to  difficulty  and  frequency 
encountered.  No  attempt  was  made  to  drive  the  experts 
opinion  on  these  measures  to  consensus.  Table  7  depicts  the 
grading  scale  used. 

The  experts  used  the  numerical  scale  listed  in  Table  7 
to  grade  the  complexity  of  the  six  cases  they  selected.  The 
case  with  the  highest  combined  difficulty  and  frequency 
score  was  selected  as  the  complex  test  case  and  the  case 
with  the  lowest  combined  difficulty  and  frequency  score  was 
selected  as  the  simple  test  case.  Additionally,  the  case 
with  the  second  lowest  combined  difficulty  and  frequency 
score  was  selected  as  an  example  case  to  be  used  during  the 
pre-test  demonstration.  The  experts  then  ran  the  two 
selected  cases  with  MARS  to  insure  the  cases  were  within  the 
limits  of  MARS’  capabilities. 
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Table  7 

Case  Complexity  Grading  Scale  [3:131] 


Di f  f i cul ty 


Descr i ptor  Grade 

Very  difficult  5 

Di f  f icul t  4 

Moderately  difficult  3 

Easy  2 

Very  easy  1 


Frequency 


Descriptor  Grade 

Very  Infrequently  Encountered  5 

Infrequently  Encountered  4 

Moderately  Encountered  3 

Frequently  Encountered  2 

Very  Frequently  Encountered  1 


Step  3  2.  Test  Key  Development.  The  development  of  a 
test  key  was  completed  before  actual  testing  occurred  in 
order  to  make  the  effectiveness  performance  evaluation  as 
objective  as  possible.  The  four  developmental  experts  were 
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asked  to  manually,  without  the  assistance  of  MARS,  complete 
the  simple  test  case  and  the  complex  test  cases  selected 
previous  1 y . 

The  formally  scored  parameter  was  the  quantity  listed 
in  block  15  of  the  Form  47,  on-hand  assets.  Four  other 
parameters  were  informally  checked  to  determine  procedural 
accuracy.  Concensus  regarding  the  correct  responses  for  the 
test  case  parameters  was  achieved  through  discussion.  Table 
8  depicts  the  test  key  used  to  match  the  scored  parameter, 
on-hand  assets,  with  the  quality  of  answer  categories:  very 
poor;  poor;  mediocre;  good;  very  good. 

Testing  facilitiy  requirements  and  a  planned  testing 
schedule  were  also  developed  during  this  step.  The  request 
for  testing  facilities  and  participant  scheduling  is 
included  in  Appendix  E. 

Step  4^  Test  Admin istration.  The  test  was  conducted 
at  Warner  Robins  Air  Logistics  Center.  A  single,  between- 
subjects  experiment  was  conducted  using  a  two-by-two-bv- 
three  factorial  design.  Prior  to  the  actual  testing,  test 
subjects  were  asked  to  complete  a  questionnaire  The 
questionnaire  requested  the  test  subjects  to  report  their 
level  of  computer  experience  and  their  level  of  confidence 
in  using  a  computer.  This  information  was  considered  during 
the  analysis  of  the  efficiency  results.  A  copv  of  the 
questionnaire  completed  by  the  test  subjects  is  included  i  r. 
Appendix  G.  Written  test  directions  were  also  provided 


prior  to  testing-  A  copy  of  the  test  directions  is  included 
1 n  Append i x  F . 

The  test  subjects’  performance  was  measured  against 
either  the  simple  or  the  complex  case.  The  subjects  were 
given  a  five  to  ten  minute  introduction  to  expert  systems 
using  the  example  case  selected  during  Step  2.  Forty 
subjects  from  the  novice  group,  individuals  with  less  than 
one  year  of  experience  as  an  inventory  manager,  were 
scheduled  for  testing  by  the  senior  Materiel  Management 
staff.  Forty  subjects  from  the  journeyman  group, 
individuals  with  one  to  four  years  experience,  were 
scheduled  for  testing  by  the  senior  staff.  Forty  subjects 
from  the  advanced  group,  individuals  with  more  than  four 
vears  experience  as  an  inventory  manager,  were  scheduled  for 
testing  b v  ‘.be  senior  staf:  A  total  of  94  sub  -  sets  out  of 

the  1  2  ■?■  scheduled  f  :  r  testing  attended  the  testing  sessions 
a  *  Warner  Robins  A  L  ■ ' 

Each  sufc’ec*  received  a  score  between  0  and  100  points 
f  r  the  *  e  s  ‘  -  t  s  e  :  .  m  p  .  ■»  *  e  .1  The  score  was  determined  by 

”,  i  a  r  .  n  »*  ‘be  *es*  sut  ect  s  answers  t  :  the  test  key  scoring 
;arar*-*ers.  T  a :  .  e  •  .a.-.  :  a  p ;  .  v  .  ■  *’  ;■  ,  r  t  values  as  described 

■  r  T  a  b  1  e  y  Analysis  of  the  94  t  e  s  »  subjects  scores  was 
used  *  r  determine  .  f  inventory  manager  performance  was 
i  mpr  v«  o  t  r.  r  us  b  •  be  use  ;  t  the  MARS  expert  s  vs  tern. 

T  b  e  time  e  a  '  h  s  u  t  ■ e c t  took  t  c  complete  the  test  case 
was  re-.-rded  Time  was  measured  from  the  moment  the  subject 


Table  8 


Test  Key  Scoring  Parameters 


• 

Actual  Error  Quantity  of  Computed 
On-hand  Assets,  Simple  Case 

Quality  of  Answer 

8  or  greater 

Very  Poor 

6  to  7 

Poor 

4  to  5 

Mediocre 

2  to  3 

Good 

0  to  1 

Very  Good 

Actual  Error  Quantity  of  Computed 
On-hand  Assets,  Complex  Case 

Quality  of  Answer 

21  or  greater 

Very  Poor 

16  to  20 

Poor 

•  11  to  15 

Mediocre 

6  to  10 

Good 

t 

0  to  5 

Very  Good 

actually  began  the  test  case  until  the  subject  finished  the 
test  case.  The  test  subjects  were  unaware  that  the  test  was 
being  timed. 

The  statistical  significance  of  the  effects  of  task 
method,  user  experience,  and  type  of  case  on  the  two 
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dependent  variables'  effectiveness  test  scores  and  time  to 
completion,  were  assessed  using  a  univariate  General  Linear 
Model  (GLM)  procedure  in  conjunction  with  the  SAS  computer 


Table  9 

Assigned  Point  Values  for  Test  Subject  Answers 


Qual i ty 

Point 

of  Answer 

Value 

Very  Poor 

0  points 

Poor 

25  points 

Mediocre 

50  points 

Good 

75  points 

Very  Good 

10.0  points 

analysis  software  program.  -GLM  procedures  use  the  method  of 
least  squares  to  fit  general  linear  models  to  the  data  and 
perform  an  analysis  of  variance.  GLM  procedures  are 
especially  suited  to  unbalanced  data  where  the  cell  sizes  of 
the  factorial  experiment  are  not  equal.  Type  III  test 
effects,  referred  to  as  partial  sum  of  squares,  were 
analyzed  as  suggested  by  the  SAS  manual  [39:466].  Hardware, 
software,  and  user  system  familiarity,  were  considered 
factors  that  affected  the  time  to  completion  of  the  test 


subjects  using  the  expert  system  to  complete  the  cases. 


Additionally,  those  test  subjects  who  used  MARS  were 


asked  to  complete  a  qualitative  critique  of  the  expert 
system.  The  critiques  were  summarized  but  no  formal 
analysis  of  the  critiques  was  attempted.  A  copy  of  the 
critique  form  used  and  some  representative  examples  of  the 
critiques  received  are  included  in  Appendices  K  and  L 
respectively . 

Summary 

This  chapter  described  the  methodological  approach, 
adapted  from  a  1986  study  by  Allen,  used  to  conduct  this 
research  project.  The  research  design  incorporated  lessons 
learned  from  previous  expert  system  development  projects. 
Early  and  continued  involvement  of  the  intended  users  of  the 
expert  system,  multiple  experts,  case  analysis,  and  an 
objective  approach  to  the  evaluation  of  the  expert  system's 
performance,  were  key  lessons  that  were  used. 

An  expert  system  to  assist  inventory  managers  with  the 
asset  reconciliation  task  was  developed.  An  experiment  was 
conducted  to  determine  if  use  of  the  expert  system  could 
enable  managers  to  make  more  effective  and  efficient 
decisions.  The  results  of  the  performance  testing  is  the 
subject  of  the  next  chapter. 


Overview 


IV.  Results  and  Other  F indings 


This  chapter  presents  the  results  of  the  General  Linear 
Model  analysis  for  the  effectiveness/efficiency  experiment. 
As  discussed  earlier  in  the  research  methodology  chapter, 
data  was  collected  during  the  e f f e c t i venes s / e f f i c 1 ency 
testing  sessions.  The  data  were  then  submitted  to  a 
statistical  analysis  using  a  fixed  effect,  t wo -by - t wo - by  - 
three  factorial  model  in  conjunction  with  a  computer 
analysis  software  package  called  SAS.  The  study  being 
validated  treated  the  subject’s  experience  and  confidence  in 
interacting  with  a  computer  as  continuous  variables  and  used 
the  data  as  covariates  in  the  analysis.  The  previous  study 
found  the  covariates  did  not  explain  any  significant  amount 
of  variance  and  were  eliminated  from  the  final  anlysis.  In 
order  to  more  closely  follow  the  study  being  validated, 
the  subject's  experience  and  confidence  interacting  with  a 
computer  were  not  considered  in  the  SAS  analysis. 

A  univariate  General  Linear  Model  (GLM)  procedure  was 
used  to  conduct  an  analysis  of  variance  for  both  the 
effectiveness  data  and  the  efficiency  data.  An  alpha  level 


o  f 

.05 

was  used 

to  j  ud g e 

significance.  An  alpha 

level 

o  f 

.  10 

was  used 

to  judge 

'marginal"  significance. 

The 

objective  of  the  analysis  was  to  determine  the  magnitude  and 
direction  of  any  change  in  total  performance  caused  by  the 
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use  of  the  expert  system*  Interaction  effects  were  analyzed 
to  determine  if  relationships  between  the  effectiveness/ 
efficiency  model  variables  existed. 

The  chapter  first  reviews  the  effectiveness  performance 
scores  followed  by  a  review  of  the  efficiency  performance 
scores.  The  final  section  of  the  chapter  presents  other 
findings  from  the  research  for  which  no  formal  research 
questions  had  been  established  in  advance. 

Ef  f ectiveness  Scores 

The  GLM  procedures  analysis  did  not  reflect  any  signif¬ 
icant  three-way  interaction  of  experience  by  difficulty  by 
method.  A  marginally  significant  two-way  interaction  of 
difficulty  by  method  was  noted.  The  expert  system  improved 
performance  on  the  complex  case  more  than  on  the  simple 
case.  The  GLM  analysis  reflected  significant  main  effects 
of  both  difficulty  and  method  main  effects.  The  SAS  analyses 
of  effectiveness  scores  are  shown  in  Tables  10  and  11. 

Pi f  f icul tv  by  Method  I n ter ac t i on  Table  12  presents  the 
aggregate  effectiveness  means  for  the  difficulty  by  method 
interaction.  The  table  reveals  a  5  percent  improvement  in 
the  scores  of  item  managers  using  the  expert  system  for  the 
simple  case  and  a  37.92  percent  improvement  for  those  item 
managers  using  the  expert  system  to  complete  the  complex 
case.  The  overall  effectiveness  of  item  managers  assisted 
by  MARS  was  increased  by  19.45  percent. 
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Table  10 


i 

i  ANALYSIS  OF  ASSET  RECONCILIATION  TEST  CASE  EFFECTIVENESS 


GENERAL  LINEAR  MODELS  PROCEDURE 


CLASS 

LEVEL 

INFORMATION 

CLASS 

LEVELS 

VALUES 

EXPERIENCE* EXPER 

3 

ADVANCED  JOURNEYMAN  NOVICE 

DIFFICULTY=DIFFICUL 

2 

COMPLEX  SIMPLE 

METHOD 

2 

EXPERT-SYSTEM  MANUAL 

NUMBER  OF  OBSERVATIONS  IN  DATA  SET  =  94 


DEPENDENT  VARIABLE:  EFFECTIVENESS  SCORE 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

F  VALUE 

MODEL 

1  1 

32465.66827085 

295 1 . 42438826 

1 . 63 

ERROR 

82 

148764.38492063 

1814- 19981611 

PR  >  F 

CORRECTED 

TOTAL 

93 

181230.05319149 

0 . 1064 

R-SQUARE  C 

0.179141  64 . 

.  V. 

8386 

ROOT  MSE 

42 . 59342456 

EFFECTIVENESS 

MEAN 

65 . 69148936 

SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

EXPER 

2 

4690 . 69135587 

1  .  29 

0 . 2800 

DIFFICUL 

1 

9312. 34360469 

5.13 

0 . 026  1 

METHOD 

1 

9*69 . 98732090 

5.11 

0 . 0264 

EXPER* DIFFICUL 

2 

440 . 34083885 

0.12 

0 . 8859 

EXPER* METHOD 

2 

2604 . 64859431 

0 . 72 

0 . 4908 

DIFFICUL “METHOD 

1 

6263 . 65663569 

3 . 45 

0 . 0667 

EXPER»DIFFI C  * METHOD 

2 

9 . 70176745 

0 . 00 

0 . 9973 
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Table  11 


TEST  CASE  EFFECTIVENESS  MEANS 


DIFFICUL 

METHOD 

N 

EFFECTIVNESS 

COMPLEX 

EXPERT 

24 

72 . 9166667 

COMPLEX 

MANUAL 

20 

35 . 0000000 

SIMPLE 

EXPERT 

25 

77 . 0000000 

SIMPLE 

MANUAL 

25 

72 . 0000000 

EXPER 

DIFFICUL 

METHOD 

N 

EFFECTIVENESS 

CHANGE 

ADVANCED 

COMPLEX 

EXPERT 

8 

8 1 . 2500000 

> 

4  1.25 

ADVANCED 

COMPLEX 

MANUAL 

5 

40 . 0000000 

ADVANCED 

SIMPLE 

EXPERT 

9 

88 . 8888889 

> 

8 . 88 

ADVANCED 

SIMPLE 

MANUAL 

10 

80 . 0000000 

JOURNEYM' 

COMPLEX 

EXPERT 

8 

71 . 8750000 

> 

46 . 87 

J0URNEYM 

COMPLEX 

MANUAL 

8 

25 . 0000000 

JOURNEYM 

SIMPLE 

EXPERT 

10 

70 . 0000000 

> 

14.45 

JOURNEYM 

SIMPLE 

MANUAL 

9 

55 . 5555556 

NOVICE 

COMPLEX 

EXPERT 

8 

65.6250000 

> 

22.76 

NOVICE 

COMPLEX 

MANUAL 

7 

42.8571429 

NOVICE 

SIMPLE 

EXPERT 

6 

70 . 8333333 

> 

-12.50 

NOVICE 

SIMPLE 

MANUAL 

6 

83 . 3333333 

EXPER 

METHOD 

N 

EFFECTIVENESS 

ADVANCED 

EXPERT 

17 

85 . 294  1  176 

ADVANCED 

MANUAL 

15 

66 . 6666667 

JOURNEYM 

EXPERT 

18 

70 . 8333333 

JOURNEYM 

MANUAL 

17 

4  1  .  1764706 

NOVICE 

EXPERT 

14 

67 . 8571429 

NOVICE 

MANUAL 

13 

6  1  .  5384615 

*  CHANGE  represents  the  increase  (♦)  or  decrease  (-)  in  the 
effectiveness  of  item  managers  from  the  same  experience 
group  when  assisted  by  MARS  on  the  same  type  of  case. 
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Table  12 
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V 

>*; 

i* 
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6 
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vf  , 
\*x 


Mean  Total  Effectiveness  Scores,  Difficulty  by  Method 


Case 

Solution  Method 

Di f  f i cul ty 

Manual 

Expert  System 

Both 

Methods 

Simple 

72 . 00 

77 . 00 

74 . 50 

(25) 

(25) 

(50) 

Complex 

35.00 

79.92 

55 . 68 

(20) 

(24) 

(44) 

Both  Cases 

55 . 55 

75 . 00 

65 . 69 

(45) 

(49) 

(94) 

Ef  f iciencv  Scores 

GLM  procedures  were  run  using  the  time  to  completion 
data  accumulated  during  testing.  No  significant  three-way 
interaction  of  experience  by  difficulty  by  method  was 
reflected  in  the  GLM  analysis.  A  significant  two-way 
interaction  was  noted  for  experience  by  method.  A 
marginally  significant  two-way  interaction  of  difficulty  by 
method  was  also  noted.  The  results  demonstrated  significant 
main  effects  of  both  difficulty  and  method.  Results  of  the 
SAS  analyses  are  listed  in  Tables  13  and  14. 
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Difficulty  by  Method  Interaction.  The  GLM  procedures 
revealed  a  significant  two-way  interaction  between  the  case 
difficulty  and  task  method  treatments.  Table  15  depicts  the 
aggregate  means  for  this  efficiency  test  interaction.  The 
overall  time  to  completion  for  item  managers  assisted  by  the 
expert  system  increased  by  10.67  minutes.  On  complex  cases, 
the  time  to  completion  for  item  managers  assisted  by  MARS 
increased  6.31  minutes.  On  simple  cases,  the  time  to 
completion  for  item  managers  assisted  by  MARS  increased  by 
12.48  minutes. 

Exper l ence  by  Me thod  I nterac t i on .  GLM  procedures  also 
revealed  significance  in  this  two-way  interaction.  Table  16 
describes  the  aggregate  means  for  this  interaction.  Time  to 
completion  decreased  as  experience  level  increased  for  all 
test  cases  completed  either  manually  or  with  MARS.  The 
average  time  to  completion  for  novice  item  managers  assisted 
by  MARS  increased  15.96  minutes.  The  average  time  to 
completion  for  journeyman  item  managers  assisted  by  MARS  was 
increased  by  12.19  minutes.  The  average  time  to  completion 
for  advanced  item  managers  assisted  by  MARS  was  increased  by 
4.79  minutes  overall.  The  largest  increase  in  the  average 
time  to  completion,  20.83  minutes,  occurred  when  novice  item 
managers  were  assisted  by  MARS  on  the  simple  case.  The 
advanced  item  manager  time  to  completion  was  reduced  by  an 
average  of  4.75  minutes  when  assisted  by  MARS  on  the  complex 
case  . 
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Table  13 


ANALYSIS  OF  ASSET  RECONCILIATION  TEST  CASE  EFFICIENCY 

GENERAL  LINEAR  MODELS  PROCEDURE 
CLASS  LEVEL  INFORMATION 
CLASS  LEVELS  VALUES 


EXPERIENCE=EXPER  3  ADVANCED  JOURNEYMAN  NOVICE 


DIFFICULTY=DIFFICUL  2  COMPLEX  SIMPLE 

METHOD  2  EXPERT-SYSTEM  MANUAL 

NUMBER  OF  OBSERVATIONS  IN  DATA  SET  =  94 


DEPENDENT  VARIABLE:  TIME  TO  COMPLETION 


SOURCE 

DF 

SUM  OF  SQUARES 

MEAN  SQUARE 

F  VALUE 

MODEL 

1  1 

17599 .89956096 

1599.99086918 

14.20 

ERROR 

82 

9239.21746032 

112.67338366 

PR  >  F 

CORRECTED 

TOTAL  93  26839.11702128  0.0001 


R-SQUARE  C.V.  ROOT  MSE  TIME  MEAN 

0.655756  26.1681  10.61477196  40.56382979 


SOURCE 

DF 

TYPE  III  SS 

F  VALUE 

PR  >  F 

EXPER 

2 

436 . 44628583 

1 . 94 

0 . 1507 

DIFFICUL 

1 

12570 . 07187820 

111.56 

0 . 0001 

METHOD 

1 

2017 . 44360942 

17.91 

0 . 0001 

EXPER* DIFFICUL 

2 

229 .81284527 

1 . 02 

0 . 3652 

EXPER* METHOD 

2 

1012.41571632 

4 . 49 

0.0141 

DIFFICUL»METHO 

1 

326 . 87757168 

2 . 90 

0 . 0923 

EXPER*DIFFIC»METHOD 

2 

59.89964591 

0 . 27 

0 . 7672 
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Table  14 


GENERAL  LINEAR  MODELS  PROCEDURE 


EFFICIENCY 

MEANS 

EXPER 

DIFFICUL 

METHOD 

N 

TIME 

# 

CHANGE 

ADVANCED 

COMPLEX 

EXPERT 

8 

50. 2500000 

> 

-4 . 75 

ADVANCED 

COMPLEX 

MANUAL 

5 

55 . 0000000 

ADVANCED 

SIMPLE 

EXPERT 

9 

27 . 2222222 

> 

4 . 82 

ADVANCED 

SIMPLE 

MANUAL 

10 

22 . 4000000 

JOURNEYM 

COMPLEX 

EXPERT 

a 

57 . 2500000 

> 

11.00 

JOURNEYM 

COMPLEX 

MANUAL 

8 

46 . 2500000 

JOURNEYM 

SIMPLE 

EXPERT 

10 

38 . 5000000 

\ 

14.17 

JOURNEYM 

SIMPLE 

MANUAL 

9 

24 . 3333333 

NOVICE 

COMPLEX 

EXPERT 

8 

61 . 0000000 

> 

10.71 

NOVICE 

COMPLEX 

MANUAL 

7 

50 . 2857143 

NOVICE 

SIMPLE 

EXPERT 

6 

43 . 3333333 

> 

20 . 83 

NOVICE 

SIMPLE 

MANUAL 

6 

22 . 5000000 

DIFFICUL 

METHOD 

N 

TIME 

COMPLEX 

EXPERT 

24 

56 . 1666667 

COMPLEX 

MANUAL 

20 

49 . 8500000 

SIMPLE 

EXPERT 

25 

35 . 6000000 

SIMPLE 

MANUAL 

25 

23 . 1200000 

EXPER 

DIFFICUL 

N 

TIME 

ADVANCED 

COMPLEX 

13 

52.0769231 

ADVANCED 

SIMPLE 

19 

24 . 6842105 

JOURNEYM 

COMPLEX 

16 

51 . 7500000 

JOURNEYM 

SIMPLE 

19 

31 . 7894737 

NOVICE 

COMPLEX 

15 

56 . 0000000 

NOVICE 

SIMPLE 

12 

32 . 9166667 

EXPER 

METHOD 

N 

TIME 

ADVANCED 

EXPERT 

17 

38 . 0588235 

ADVANCED 

MANUAL 

15 

33 . 2666667 

JOURNEYM 

EXPERT 

18 

46 . 8333333 

JOURNEYM 

MANUAL 

17 

34 . 6470588 

NOVICE 

EXPERT 

14 

53 . 4285714 

NOVICE 

MANUAL 

13 

37. 4615385 

*  CHANGE  represents  the  increase  (♦)  or  decrease  (-)  in  the 
efficiency  of  item  managers  from  the  same  experience 
group  when  assisted  by  MARS  on  the  same  type  of  case. 


Table  15 


Mean  Total  Efficiency  Scores,  Difficulty  by  Method 


Case 

Di f  f i cul ty 

Solution  Method 

Manual 

Expert  System 

Both 

Methods 

S i mp 1 e 

23.12 

35 . 60 

29 . 36 

(25) 

(25) 

(50) 

Complex 

49 . 85 

56  .  16 

53 . 29 

(20) 

(24) 

(44) 

Both  Cases 

35 . 00 

45 . 67 

40 . 56 

(45) 

(49) 

(94) 

Table  16 

Mean  Total  Efficiency  Scores,  Experience  by  Method 


Solution 

Experience  Level 

Method 

A1  1 

Novice 

Journeyman 

Advanced 

Levels 

Manual 

37 . 46 

34 . 64 

33 . 26 

35 . 00 

(13) 

(17) 

(15) 

(45) 

Expert  System 

53 . 42 

46 . 83 

38 . 05 

45.67 

(14) 

(18) 

(17) 

(49) 

Both  Solutions 

45 . 73 

40.91 

35 . 80 

40 . 56 

(27) 

(35) 

(32) 

(94) 

Other  F indings 

Three  findings,  for  which  no  formal  research  questions 
had  been  established  in  advance,  were  discovered  during  the 
conduct  of  this  research. 

The  first  finding  concerned  asset  reconciliation 
policy.  Several  policy  operating  procedures  were  found  to 
be  in  need  of  review.  The  developmental  experts  questioned 
the  value  of  the  current  worldwide  inventory  procedures  as 
written  in  AFM  67-1.  Revisions  to  the  Form  47  were  also 
recommended.  The  developmental  experts  suggested  removing 
the  ‘On-loan  (Bailments)"  line  from  the  Form  47  and  chang¬ 
ing  the  ‘Transfers  to  R  &  M‘  line  to  ‘Transfers  to  DPDO". 

The  second  finding  refers  to  the  use  of  Exercise  Three 
during  the  knowledge  engineering  effort.  The  methodology 
used  during  the  exercise  proved  to  be  especially  helpful  m 
stimulating  a  detailed  discussion  regarding  asset  reconcil¬ 
iation  procedures.  The  exercise  focused  the  thinking  of  the 
developmental  experts  and  gave  them  a  better  understanding 
of  where  the  knowledge  engineering  sessions  were  leading. 

The  third  finding  was  determined  through  the  analysis 
of  the  subject  questionnaires  which  were  distributed  prior 
to  the  effectiveness/efficiency  testing  sessions.  The  com¬ 
puter  literacy  of  the  test  subjects  was  self  reported. 
Sixty-seven  percent  of  the  subjects  tested,  reported  either 
no  computer  experience  or  limited  computer  experience. 


Sixty-one  percent  of  the  subjects  tested  described  them- 


selves  as  having  either  no  confidence  or  limited  confidence 


in  their  use  of  computers. 

Summary 

Chapter  IV  presented  the  results  and  other  findings  of 
this  research  project.  The  effectiveness  of  item  managers 
who  were  assisted  by  the  expert  system  was  higher  than  the 
effectiveness  of  item  managers  who  performed  the  test 
manually.  A  19.45  percent  improvement  in  the  overall 
effectiveness  of  test  subjects  assisted  by  MARS  was  demon¬ 
strated.  On  complex  cases,  test  subject  effectiveness  in¬ 
creased  by  37.92  percent. 

The  time  to  completion  of  item  managers  who  were 
assisted  by  the  expert  system  was  longer  than  the  time  to 
completion  of  item  managers  who  performed  the  test  manually. 
An  increase  of  10.67  minutes  for  the  overall  time  to 
completion  for  test  subjects  assisted  by  MARS  was  observed. 
On  complex  cases,  the  time  to  completion  for  test  subjects 
assisted  by  MARS  was  increased  by  6.31  minutes. 

Chapter  V  will  present  the  conclusions  drawn  from  these 
results  and  other  findings. 
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V. 


Summary ,  Conclusions .  and  Recommendations 


Overview 

The  purpose  of  this  chapter  is  to  provide  an 
overview  of  the  entire  research  project  with  emphasis 
accorded  to  the  research  conclusions  and  the  recommendations 
for  further  study.  The  first  section  summarizes  the  primary 
objective  of  the  research  effort,  the  research  methodology, 
and  the  research  findings.  The  second  section  discusses 
conclusions  drawn  from  the  research  and  assesses  the 
contributions  made  by  the  study.  The  final  section  provides 
recommendations  for  future  research. 

Summary  of  Research  Effort 

This  section  will  briefly  review  the  research 
objective,  the  methodology  applied  to  achieve  the  research 
objective,  and  the  research  findings. 

Research  Objective.  The  objective  of  this  research 
was  to  validate  a  previous  research  finding  that  documented 
a  15  percent  increase  in  the  effectiveness  of  AFLC  inventory 
managers  when  assisted  by  an  expert  system.  The  research 
was  conducted  to  extend  prior  expert  system  research  by 
developing  an  expert  system  for  a  different  task.  The 
development  of  an  expert  system  for  asset  reconciliation 
that  increased  the  effectiveness  of  item  managers  using  the 
expert  system  by  19.45  percent  validated  the  previous 
finding  and  extended  the  research  of  the  previous  study. 


Ultimately,  the  research  was  undertaken  to  determine  if  an 
expert  system  for  asset  reconciliation  could  improve 
inventory  management  procedures  and  potentially  create 
savings  for  the  Government  while  alleviating  manpower 
shortages . 

Research  Methodology.  The  methodological  approach 
to  the  research  effort  involved  a  twelve-step  process  that 
was  divided  into  three  general  phases:  Problem  Selection: 
Knowledge  Engineering:  and  Performance  Measurement.  The 
objective  of  the  problem  selection  phase  was  to  choose  a 
problem  to  be  addressed  by  the  expert  system  and  to  select 
domain  experts  whose  knowledge  would  be  encoded  into  the 
expert  system  to  solve  the  chosen  problem.  The  purpose  of 
the  knowledge  engineering  phase  of  the  research  was  to 
extract  knowledge  from  the  selected  asset  reconciliation 
experts  in  order  to  create  a  knowledge  base  for  the 
prototype  expert  system.  The  final  phase  of  the  research, 
performance  evaluation,  was  conducted  to  determine  if  the 
developed  prototype  expert  system  enhanced  the  productivity 
of  inventory  managers  using  the  system. 

Research  Findings.  The  findings  of  the  research 
were  grouped  according  to  effectiveness  and  efficiency.  The 
overall  effectiveness  of  item  managers  assisted  by  the 
expert  system  was  increased  by  19.45  percent.  The  overa'l 
efficiency,  i.e.  time  to  completion  of  the  test  cases,  of 
item  managers  assisted  by  the  expert  system  was  increased  by 
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10.67  minutes.  Three  other  findings,  for  which  no  formal 
research  questions  had  been  established  in  advance,  were 
also  discovered.  These  findings  included  suggested  changes 
to  asset  reconciliation  policy,  the  successful  use  of  an 
expert  simulation  procedure  during  knowledge  engineering 
sessions,  and  that  67  percent  of  the  item  managers  tested 
reported  either  no  computer  experience  or  limited  computer 
exper i ence . 

Conclusions 

The  conclusions  discussed  in  this  section  are  related 
to  the  five  research  questions  addressed  by  this  study. 

Research  Question  1_:  Is  asset  reconciliation  a 

suitable  problem  for  application  of  an  expert  system9 

Conclusion  1:  Asset  reconciliation  was  determined  to 

be  a  suitable  problem  for  application  of  an  expert  system. 
Asset  reconciliation  met  the  problem  selection  criteria 
described  in  Chapter  II.  The  characteristics  of  the  asset 
reconciliation  problem  demonstrated  that  expert  system 
application  was  possible,  justified,  and  appropriate.  The 
effectiveness  of  item  managers  performing  a  complex  asset 
reconciliation  test  case  with  the  assistance  of  MARS  was 
increased  by  37.92  percent  over  the  effectiveness  of  those 
item  managers  completing  the  complex  test  case  without 
MARS.  The  increased  effectiveness  of  the  item  managers 
supports  the  assumption  that  the  asset  reconciliation 
task  is  indeed  suited  for  application  of  an  expert  system. 
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The  positive  critiques  received  from  the  subjects  who  were 
assisted  by  MARS  during  testing  suggests  the  potential  for 
widespread  user  acceptance  is  high.  The  strong  user 
acceptance  also  suggests  that  an  expert  system  for  the  asset 
reconciliation  problem  task  was  a  suitable  application. 

Research  Question  2.:  Can  the  knowledge  needed  to  solve 
the  selected  problem  be  extracted  from  the  experts9 

Conclusion  2:  The  finding  that  test  subjects  assisted 


the  expert  system  had  higher  effectiveness  scores  for 


both  the  simple  and  complex  cases  suggests  that  the 


knowledge  reauired  to  solve  an  asset  reconciliation  problem 


was  extracted  from  the  human  experts.  Item  managers  using 
the  expert  system  remarked  that  the  expert  system  broadened 
their  understanding  of  the  asset  reconciliation  process. 
One  technique  proved  especially  productive  in  extracting  the 
required  knowledge  from  the  developmental  experts.  The 
Nominal  Group  Technique  was  used  successfully  throughout  the 
knowledge  engineering  process.  The  Nominal  Group  Technique 
served  to  structure  the  knowledge  engineering  sessions  and 
to  surface  underlying  sources  of  disagreement  between  the 


developmental  experts. 


Conclusion  3:  The  use 


a  previous  knowledge 


engineering  experiment,  that  mimics  computer  interaction  b: 


having  several  domain  experts  act  as  the  expert  system  while 


another  domain  expert  acts  as  the  novice  user  of  the  expert 


system,  was  applied  successful!' 


the  MARS  knowledge 
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engineering  effort-  The  simulation  of  the  interaction 
between  an  expert  system  and  a  novice  user  proved  to  be 
highly  beneficial  in  focusing  the  questions  and  answers 
suggested  by  the  developmental  experts  for  an  asset 
reconciliation  computation. 

Conclusion  4:  A  conclusion  in  the  study  being 
validated  stated.  'multiple  experts  should  be  used  to 
discover  the  heuristics  for  complex  cases',  this  conclusion 
was  definitely  applicable  to  the  MARS  knowledge  engineering 
effort.  The  range  of  opinions  and  ideas  offered  by  multiple 
experts  is  needed  to  account  for  the  special  cases  and 
unique  situational  factors  that  can  occur  during  case 
analysis . 

Research  Question  3:  Can  the  rules  needed  to  perform 
an  asset  reconciliation  be  encoded  using  the  Teknowledge  M.  1 
software  program'’ 

Conclusion  5:  The  Teknowledge  M. 1  software  program  was 
capable  of  incorporating  the  rules  needed  to  perform  the 
limited  task  of  asset  reconciliation.  The  expert  system 
prototype  which  was  developed,  the  Manager’s  Asset 
Reconciliation  System  (MARS)  contained  150  rules  and 
demonstrated  the  capability  of  reconciling  numerous  types  of 
assets  under  a  range  of  circumstances.  The  increased 


effectiveness  of  item  managers  using  MARS  demonstrated  their 
greater  understanding  of  the  asset  reconciliation  process. 
The  Teknowledge  M.l  software  program  was  proven  to  be 


capable  of  encoding  the  required  rules  and  facts  needed  for 
the  asset  reconciliation  task. 

Domain  expert  critiques  of  MARS,  both  verbal  and 
written,  suggested  modifications  for  the  expert  system.  A 
'fast  version"  of  the  system  which  bypasses  much  of  the 
advice  offered  by  the  MARS  prototype  was  suggested.  The 
assumption  could  be  made  that  a  quicker  consultation  would 
be  an  important  benefit  in  convincing  potential  users  of  the 
system’s  worth.  The  "fast  version"  would,  theoretically,  be 
for  k  Dwledgeable  item  managers  who  do  not  require  a  full 
explanation  of  all  the  steps  in  the  asset  reconciliation 
process.  The  capability  of  choosing  either  a  fast  version, 
slow  version,  or  some  combination  of  the  two  could  reduce 
the  consultation  time  of  the  system  dramatically.  However, 
the  choice  to  bypass  information  might  also  reduce  the 
effectiveness  of  the  system  by  allowing  users  to  make  more 
errors.  The  M.l  software  program  can  incorporate  the 
variable  versions  of  MARS  suggested. 

Storing  historical  data  in  the  expert  system  was 
another  suggested  modification  to  MARS.  Storing  past 
quarters  of  relevant  item  information  will  speed  up  the 
asset  reconciliation  process.  A  reduction  in  computational 
errors  is  assumed  due  to  the  elimination  of  data  transferal 
errors  and  simple  mathematical  errors.  The  suggested 
modification  can  be  achieved  by  loading  the  historical  data 
into  the  memory  of  MARS. 


The  need  for  a  greater  capability  to  change  asset 
quantities  once  they  are  entered  in  MARS  was  another 
suggestion  offered  by  the  domain  experts.  The  current 
limited  correction  capability  of  MARS  can  be  improved  by  the 
modification  of  current  rules  in  MARS  and  by  the  inclusion 
of  additional  rules.  The  M. 1  software  package  is  capable  of 
incorporating  the  modifications  and  additions  required  to 
comply  with  all  the  improvements  suggested  in  the  test 
subjects  critiques. 

Conclusion  6:  The  development  of  an  expert  system  does 
not  necessarily  require  extensive  time  and  large  monetary 
outl ays .  The  implementation  of  expert  systems  using 
relatively  low-cost  microcomputers  appears  to  be  a  viable 
option  in  the  Materiel  Management  arena  of  operations.  The 
initial  prototype  was  developed  in  only  three  months  using 
the  M.  1  software  program  hosted  on  a  microcomputer.  The 
development  of  the  MARS  prototype  cost  approximately 
$10,000.  This  rough  approximation  includes  travel  and 
manpower  expenses  for  the  developmental  experts,  the 
knowledge  engineer,  and  other  personnel  associated  with  the 
project.  The  expense  of  a  microcomputer,  the  M.l  software 
program,  and  a  printer  were  considered  sunk  costs  and  were 
not  included  in  the  cost  approximation. 

Research  Question  4:  How  does  the  developed  expert 
system  prototype  affect  the  productivity  of  the  using 


organization'’ 


Research  Question  4  was  divided  into  three  specific 


questions.  The  first  being:  Does  the  use  of  the  expert 
system  increase  effectiveness  by  at  least  lSX7 

Conclusion  7:  A  19.45  percent  overall  improvement  in 
the  effectiveness  of  those  item  managers  assisted  by  MARS 
seems  to  suggest  that  savings  could  be  realized  if  MARS  was 
i mpl emented.  A  5  percent  improvement  in  the  effectiveness 
scores  of  item  managers  tested  with  the  simple  case  was 
achieved.  A  simple  case  is  one  that  is  frequently 
encountered  and  has  fewer  possible  dimensions  in  the 
decision-making  scenario  than  a  complex  case.  Even  a  5 
percent  improvement  in  the  effectiveness  of  managing  assets 
could  be  transformed  into  substantial  savings.  The  MARS 
prototype  demonstrated  a  37.92  percent  improvement  in  the 
effectiveness  scores  of  those  item  managers  tested  with  the 
complex  case.  The  large  gain  in  effectiveness  suggests  that 
MARS  has  the  potential  of  improving  inventory  management 
procedures  and  saving  the  Government  considerable  financial 
resources . 

The  first  test  run  of  MARS  against  a  completed  AFLC 
Form  47,  which  had  been  thoroughly  checked  for  accuracy  by 
several  item  managers  beforehand,  discovered  the  reconciled 
asset  quantity  to  be  seven  items  in  error.  The 
inaccuracy  was  a  misrepresentation  of  inventory  assets 
equalling  nearly  *20,000.  The  enhanced  management  of 
inventory  items  not  only  reduces  the  cost  of  carrying 


inventory  but  also  improves  the  overall  readiness  of  the  Air 
Force  by  improving  the  accuracy  of  a  logisitics  support 
function.  Mismanagement  of  inventory  results  in  procuring 
items  the  Government  does  not  require;  not  procuring  items 
the  Government  does  require;  terminating  items  before  or 
after  the  items  should  be  terminated;  and  disposing  of  items 
before  or  after  the  items  should  have  been  disposed.  MARS 
should  be  further  developed  and  tested  because  of  the 
system's  tremendous  potential  for  improving  the  inventory 
management  process  and  saving  the  government  money. 

The  second  performance  question:  Does  the  use  of  the 
expert  system  reduce  the  time  required  by  the  item  manager 
to  complete  an  asset  reconciliation"? 

Conclusion  8:  The  research  findings  revealed  that  item 
managers  assisted  by  the  expert  system  took  10.67  minutes 
longer  overall  to  complete  both  the  complex  test  case  and 
the  simple  test  case.  The  poor  performance  is  attributed, 
in  part,  to  the  inflexibility  of  the  system  consultation. 
MARS  did  not  allow  the  users  to  freely  scroll  back  and  forth 
to  whatever  section  of  the  Form  47  they  wished  to  address 
during  the  consultation.  ‘Backtracking’  capabilities  were 
available  at  various  points  throughout  the  program  but  the 
lack  of  user  familiarity  with  the  expert  system  reduced  the 
effectiveness  of  this  program  feature.  The  limited 
capabilty  to  make  changes  after  item  quantities  were  entered 
and  the  relatively  advanced  wording  of  questions  presented 


by  the  expert  sytem  were  also  drawbacks  to  the  system/user 
efficiency  level. 

Subjects  tested  were  only  given  a  five  to  ten  minute 
introduction  regarding  expert  system  consultations.  The 
introduction  involved  a  step  by  step  process  through  the 
first  few  questions  of  an  example  case.  Therefore, 
virtually  all  of  the  questions  presented  to  the  subject 
during  the  test  case  were  seen  by  the  subject  for  the  very 
first  time.  The  subjects  lack  of  familiarity  with  the  MARS 
system  is  assumed  to  have  affected  efficiency  negatively. 

Conclusion  9:  Training  of  item  managers  in  computer 
procedures  and  expert  system  operations  will  increase  the 
efficiency  of  future  expert  system  applications.  A  major 
contributor  to  the  reduction  in  the  efficiency  of  MARS  was 
assumed  to  be  the  computer  skill  level  of  the  item 
mangagers.  The  self  reported  computer  experience  and 
computer  confidence  of  the  items  managers  that  participated 
in  the  testing  was  quite  low.  Sixty-seven  percent  of  the 
subjects  tested  reported  either  no  computer  experience  or 
limited  computer  experience.  Sixty -  one  percent  of  the 
subjects  tested  described  themselves  as  having  either  no 
confidence  or  limited  confidence  regarding  computer  useage. 

Two  individuals  were  afforded  the  opportuni  ty  of 
exposure  to  MARS  before  using  the  system  for  completion  of 
an  asset  reconciliation  test  case.  Both  of  these  indi¬ 


viduals  received  100  points  for  their  effectiveness  scores; 


additionally,  one  of  these  individual  recorded  the  fastest 


time  to  completion  for  her  journeyman  item  manager  group, 
while  the  other  individual  recorded  an  above  average  time  to 
completion  for  her  novice  item  manager  group.  These  results 
suggest  the  possibility  of  considerable  efficiency  gains 
after  the  expert  system  training  of  item  managers  is 
accompl ished . 

The  third  performance  related  question  was  subjective 
in  nature:  What  is  the  user’s  subjective  evaluation  of  the 
assistance  provided  by  the  expert  system  for  asset 
reconciliation9 

Conclusion  10:  The  test  subjects  thought  MARS  was  a 
good  system  with  tremendous  potential  for  improving  how  item 
managers  perform  the  asset  reconciliation  function.  The 
test  subject  critiques  of  the  system  demonstrated  positive 
feedback  and  are  included  in  Appendix  L.  The  written  and 
oral  critiques  offered  by  the  Materiel  Management  personnel 
at  WR-ALC  seem  to  suggest  that  there  is  a  real  need  for  the 
system.  The  importance  of  user  acceptance  was  stressed 
throughout  the  literature  reviewed  for  this  research 
project.  The  apparent  level  of  user  acceptance  of  MARS  was 
high.  The  positive  critiques  suggest  that  the  implementa¬ 
tion  of  such  a  system  has  a  strong  chance  for  success  if 
properly  introduced,  expanded,  and  maintained. 

Research  Question  5:  How  significant  is  experience 
level  in  determining  the  performance  level  of  item  managers 


assisted  by  the  expert  system7  Does  use  of  the  expert 
system  help  less  experienced  item  managers  more  than 
experienced  item  managers7 

Conclusion  11:  The  experience  level  of  the  item 
managers  seemed  to  have  an  impact  on  the  change  in 
effectiveness  caused  by  the  MARS  prototype.  The  experience 
by  method  interaction  for  effectiveness  was  not  found  to  be 
significant.  The  researcher  expected  the  novice  group  of 
item  managers  to  benefit  the  most  from  MARS,  followed  by  the 
journeyman  group,  with  the  advanced  group  of  item  managers 
benefiting  the  least.  The  actual  findings,  although  not 
significant,  did  not  conform  to  this  presupposed  pattern  of 
increased  benefit  as  the  experience  levels  decreased.  The 
actual  test  results  demonstrated  the  amount  of  improvement 
varied  considerably  within  experience  groups  between  the 
simple  and  complex  cases. 

The  advanced  item  managers  improved  their  effectiveness 
scores  when  using  MARS  by  41.25  percent  on  the  complex  case 
and  3.88  percent  on  the  simple  case  for  an  overall 
increase  in  effectiveness  of  18.62  percent.  Journeyman  item 
managers  improved  their  effectiveness  when  assisted  by  MARS 
46.87  percent  on  the  complex  case  and  14.45  percent  on  the 
3imple  case  for  an  overall  increase  in  e f f e c t l ven e s s  of 
29.66  percent.  Novice  item  managers  improved  their 
effectiveness  scores  by  22.76  percent  on  the  complex  case 


and  actually  reduced  their  effectiveness  scores  by  12.50 


percent  on  the  simple  case  for  an  overall  increase  in 


effectiveness  of  6.32  percent.  These  findings  suggest  MARS 
was  too  complicated  for  the  novice  item  manager  to  gam  the 
full  benefit  of  the  knowledge  encoded  in  the  expert  system. 
To  make  MARS  better  suited  for  novice  item  managers,  more 
detailed  descriptions  of  where  to  find  the  information 
needed  to  perform  an  asset  reconciliation  should  be  included 
in  the  MARS  consultation  program.  The  use  of  'novice' 
language,  i.e.  simplified  wording,  for  the  consultation 
should  improve  the  value  of  MARS  greatly. 


Table  17 

Mean  Total  Effectiveness  Scores,  Experience  by  Method 


Solution 

Experience  Level 

Method 

Novice 

Journeyman 

Advanced 

All 

Leve 1 s 

Manua 1 

6  1.53 

(13) 

4  1.17 

(  17) 

66 . 67 
(15) 

55 . 55 
(45) 

Expert  System 

67 . 85 

(14) 

70 . 83 
(  18) 

85 . 29 
(17) 

75 . 00 

(49) 

Both  Solutions 

64.81 

(27) 

56 . 43 
(35) 

75 . 56 
(32) 

65 . 69 
(94  ) 

The  large  increase  in  the  effectiveness  of  the 


journeyman  category  is  attributed  to  the  following: 

1.  The  language  of  MARS,  i.e.  the  wording  of  the 
consultation  questions,  proved  to  be  difficult  for  the 
novice  item  manager  to  interpret  but  was  not  too  difficult 
for  the  journeyman  item  manager. 

2.  The  journeyman  item  managers  had  more  room  for 
improvement  than  the  more  proficient  advanced  item  manager. 

Conclusion  12:  The  experience  level  of  the  item 
managers  had  a  significant  affect  on  the  change  in 
efficiency  of  those  item  managers  assisted  by  the  MARS 
prototype.  The  experience  by  method  interaction  for 
efficiency  was  found  to  be  significant.  The  researcher  did 
not  expect  any  increases  in  efficiency  to  be  achieved  by 
item  managers  assisted  by  MARS  because  of  their  lack  of 
familiarity  with  computer  operation  and  the  expert  system. 
If  an  efficiency  improvement  were  to  be  observed,  the 
researcher  assumed  the  time  to  completion  of  the  novice 
group  of  item  managers  would  be  reduced  the  most  when  they 
were  assisted  by  MARS.  The  researcher  also  assumed  that  the 
time  required  to  complete  a  test  case  by  the  advanced  group 
of  item  managers  would  be  affected  the  least  when  assisted 
by  MARS.  The  findings  described  in  Chapter  IV  did  not 
support  these  contentions.  Efficiency  score  firdings  demon¬ 
strated  that  MARS  improved  the  efficiency  of  advanced  item 
managers  on  complex  cases  while  all  other  treatment 


combination  times  to  completion  were  increased, 


The 


advanced  item  managers  actually  reduced  their  average  time 
to  completion  by  4.75  minutes  when  assisted  by  MARS  on  the 
complex  test  case.  The  largest  increase  in  the  amount  of 
time  required  to  complete  a  test  case  occurred  when  the 
novice  group  was  assisted  by  MARS  on  the  simple  case.  The 
novice  group  average  time  to  completion  for  the  simple  case 
was  increased  from  22.5  minutes  to  43.3  minutes  when  they 
were  assisted  by  MARS.  These  findings  seem  to  support  the 
contention  that  the  wording  of  the  questions  posed  by  MARS 
was  set  at  too  high  a  level  and  should  be  reworded  to  be 
more  easily  understood  by  novice  item  managers.  Several 
changes  to  simplify  the  system  are  already  being  implemented 
by  this  researcher.  Once  the  modifications  to  MARS  are 
completed  the  system  should  be  of  considerably  more  value  to 
novice  item  managers  in  their  daily  work  activities  and 
should  also  be  iseful  for  training. 

Conclusion  13:  Substantial  novice  input  should  be 
included  in  the  development  of  future  expert  system 
applications.  The  results  of  the  experience  by  method 
analysis  and  the  review  of  test  subject  critiques  suggest 
that  greater  novice  input  during  prototype  development 
should  broaden  the  group  of  users  that  can  successfully  use 
the  expert  system.  Required  actions  that  appear  obvious  to 
an  expert  might  be  ignored  during  the  knowledge  engineering 
process  without  the  input  of  a  less  experienced  novice. 


Recommendat i ons  for  Future  Research 


The  use  of  expert  system  technology  to  improve  inven¬ 
tory  management  operations  includes  a  wide  range  of  topics 
to  be  considered  for  further  study.  Three  suggestions  for 
future  research  include  the  development  and  evaluation  of 
new  expert  systems,  the  implementation  and  maintenance  of 
expert  systems,  and  the  extension  of  studies  regarding  MARS. 

The  inventory  management  function  offers  a  wide  range 
of  tasks  for  which  an  expert  system  could  be  developed. 
For  example,  the  Air  Force  Logistics  Command  (AFLC)  is  in 
the  process  of  developing  12  major,  integrated,  information 
syste ms  under  the  Logistics  Management  Syste  ms  ( L  MS ) 
Modernization  Program.  Research  could  be  conducted  to 
develop  and  measure  the  impact  of  an  expert  system  that 
supports  the  interaction  of  users  with  any  of  the  new  LMS 
Modernization  Programs,  i.e.  the  Requirements  Data  Bank 
(RDB)  or  the  Stock  Control  and  Distribution  system  (SC  &  D)  . 
Such  an  expert  system  would  assist  the  user  in  accessing  the 
information  system,  interpreting  the  accessed  data,  and 
applying  the  data  in  a  productive  manner.  The  application 
of  expert  systems  to  support  the  user  interface  with  LMS  is 
an  area  of  research  that  offers  tremendous  potential 
returns . 

Considerable  research  has  yet  to  be  accomplished 
regarding  the  implementation  and  maintenance  of  expert 
systems.  Post  developmental  issues  are  extremely  important 


to  the  overall  success  of  any  new  system.  Research  could 
reveal  which  procedures  result  in  the  most  successful 
introduction  of  an  expert  system  and  which  element  of  the 
using  organization  should  be  responsible  for  the  maintenance 
of  the  expert  system  once  the  system  is  in  place.  For 
example,  should  the  developed  expert  system  be  introduced  by 
the  developer  or  by  someone  in  a  leadership  role  in  the 
using  organization7  Should  the  user’s  themselves  be 
responsible  for  maintenance  of  the  expert  system  or  should 
the  responsibility  for  upkeep  be  assumed  by  a  centralized 
department  within  the  organization? 

Another  area  of  possible  investigation  involves  the 
extension  of  the  MARS  study,  for  example,  further 
performance  measurement  of  MARS.  The  evaluation  of  item 
managers  assisted  by  MARS  should  be  conducted  at  ALC’s  other 
than  Warner  Robins  ALC  in  order  to  determine  if  MARS  can  be 
applied  productively  by  item  managers  throughout  the  Air 
Force  Logistics  Command.  The  evaluation  of  item  managers 
who  have  had  adequate  practice  in  the  use  of  MARS  prior  to 
testing  should  also  be  conducted.  Development  of  an 
advanced  version  of  MARS  which  includes  other  item  manager 
functions  could  be  attempted.  The  development  of  this 
advanced  version  could  be  accomplished  using  50  percent 
novice  item  managers  and  50  percent  advanced  item  managers 
in  an  attempt  to  determine  the  most  appropriate  mix  of 
experience  required  for  expert  system  developmental 


personnel-  MARS  could  be  further  developed  to  include  the 
capability  of  automatically  accessing  all  the  required  data 
to  compute  a  Form  47  needing  human  interaction  only  for 


predetermined  exceptional  cases.  Further  analysis  of  the 
affects  of  computer  experience  and  computer  confidence  on 
expert  system  performance  could  also  be  attempted.  Fol¬ 
lowing  the  modification  and  upgrade  of  MARS,  item  manager 
performance  should  be  measured  again  to  determine  the  impact 
of  the  modifications  on  the  effectiveness  and  efficiency  of 
item  managers  assisted  by  MARS. 

Expert  systems  will  have  a  tremendous  influence  upon 
logistics  operations  in  the  future.  Considerable  research 
still  needs  to  be  done  in  support  of  expert  system 
application  to  logistics  management  functions.  The  research 
conducted  broadened  the  base  of  knowledge  regarding  expert 
system  applications  and  demonstrated  the  promise  of  expert 


system  technology  in  the  field  of  materiel  management. 


Appendix  A:  Definitions 


Artif icial  intelligence  -  The  part  of  computer  science 
concerned  with  developing  intelligent  computer  programs 
[51:10]. 

Backward-chaining.  Backchaining  -  A  reasoning  strategy 
produced  by  the  combination  of  rule-based  representation, 
modus  ponens  inference,  and  goal -directed  control.  In  this 
strategy,  a  chain  of  inferences  is  constructed  starting  with 
a  rule  whose  conclusion  would  immediately  provide  a  solution 
to  the  problem  (the  goal).  Then  the  system  works  backward, 
setting  up  the  premise  clauses  of  the  rule  as  subgoals. 
Other  rules  whose  conclusions  could  satisfy  these  subgoals 
are  then  invoked,  and  their  premises  also  become  subgoals. 
The  process  continues  working  backward  until  rules  are 
reached  whose  premises  can  be  verified  directly.  In  an 
exhaustive  backward-chaining  system,  all  possible  rules  are 
examined  to  arrive  at  a  result  [45:Q-2], 

Case  -  A  particular  problem  for  which  a  knowledge  system  can 
perform  its  task.  For  example,  a  car  diagnosis  system  might 
find  the  cause  of  a  charging  problem  in  Frank  Smith's 
Chevrolet.  Cases  are  used  to  develop,  expand,  and  evaluate 
the  performance  of  knowledge  systems  l45:G-3], 

Do ma i n  -  A  field,  discipline,  or  area  of  human  activity  or 
scholarship  such  as  mathematics,  medicine,  physics,  auto 
mechanics,  etc.  Existing  knowledge  systems  perform 
specialized  tasks  within  domains.  No  system  yet  offers 
adyice  about  an  entire  domain  [45:G-5]. 

Domain  expert  -  A  person  who  through  years  of  training  and 
experience  has  become  extremely  proficient  at  problem 
solving  in  a  particular  domain  [51:10]. 

End-user  -  The  person  who  uses  the  finished  expert  system; 
the  person  for  whom  the  system  was  developed  [51:10], 

End-User  System  -  A  knowledge  system  that  helps  or  advises  a 
user  in  a  way  that  parallels  the  way  a  human  consultant 
advises  a  client.  The  knowledge  system  may  ask  questions  of 
the  user  to  gather  information  from  which  to  infer 
appropriate  advice.  Some  consultation  systems  (including 
knowledge  systems  developed  using  M.l)  are  able  to  explain 
their  reasoning  so  that  the  user  can  judge  its  reliability 
[45:G-0] . 


Expert  -  A  human  who  can  solve  problems  or  perform  actions 
requiring  unusual  or  uncommon  skill.  The  expert’s  skills 
are  likely  to  have  evolved  slowly,  and  usually  require 
extensive  knowledge  and  lengthy  experience  [45:G-6]. 

Expert i se  -  The  set  of  capabilities  that  underlies  the  high 
performance  of  human  experts,  including  extensive  domain 
knowledge,  and  compiled  forms  of  behavior  that  afford  great 
economy  in  skilled  performance.  Expert  knowledge  is  rare, 
seldom  explicit  or  measurable,  and  difficult  to  communicate 
or  acquire.  Gained  through  experience,  long  periods  of 
training,  apprenticeship  and  observation,  it  may  be 
organized  in  subtle  and  idiosyncratic  ways  [45:G-6]. 

Expert  system  -  A  computer  program  using  expert  knowledge  to 
attain  high  levels  of  performance  in  a  narrow  problem  area 
[51:11].  A  knowledge  system  that  performs  at  or  near  the 
level  of  human  experts.  The  system  achieves  high  levels  of 
performance  on  tasks  that,  for  human  beings,  require  years 
of  special  education  and  training.  Often  the  use  of  this 
term  is  synonymous  with  knowledge  system.  However,  we  make 
the  distinction  that  an  expert  system  is  a  special  kind  of 
knowledge  system,  one  that  uses  knowledge  acquired  from  an 
expert  rather  than  from  other  sources  [45:Q-6], 

Goal  -  The  result  or  achievement  toward  which  effort  is 
directed.  In  M.  1  ,  a  goal  is  an  expression  for  which  a  value 
is  being  sought.  The  goals  of  a  consultation  are  declared 
in  an  M.l  knowledge  base  using  the  goal  meta-fact  [45:G-8]. 

Heuristic  -  A  rule  of  thumb  or  simplification  that  limits 
the  search  for  solutions  in  domains  that  are  difficult  and 
poorly  understood  [51:223. 

Inference  -  The  process  of  deriving  new  facts  from  known 
ones.  The  method  of  inference  defines  the  one-step 
derivations  that  are  legal  with  respect  to  a  given 
representation  in  a  knowledge  system.  Examples  of  inference 
methods  are  modus  ponens  and  inheritance.  Together  with 
representation  and  control,  inference  is  one  of  the  three 
dimensions  of  a  knowledge  system  [45:G-93. 

Inference  engine  -  That  part  of  a  knowledge-based  system  or 
expert  system  that  contains  the  general  problem-solving 
knowledge  [51-22]. 

Knowledge  -  Propositions  and  heuristic  rules.  Those  kinds 
of  data  that  can  improve  the  efficiency  or  effectiveness  of 
a  problem  solver.  Types  of  dimensions  such  as:  symbolic  vs 
numeric,  factual  vs  heuristic,  imperative  vs  declarative, 
deep  vs  shallow  [45:G-103. 


Kno wl ed ge  Acquis ition  -  The  extraction  and  representation  of 
knowledge  derived  from  current  sources,  especially  from 
experts.  The  activity  of  transferring  or  transforming  the 
knowledge  and  skills  from  a  human  expert  or  other  knowledge 
source  into  a  form  usable  by  a  knowledge  system.  Knowledge 
acquisition  often  involves  chunking  or  parsing  an  expert’s 
knowledge  into  a  collection  of  if-then  rules  [45:G-103. 

Knowledge-based  system  -  A  program  in  which  the  domain 
knowledge  is  explicit  and  separate  from  the  program's  other 
knowledge  [51:23]. 

Knowledge  base  -  The  portion  of  a  kno wl edge - based  system  or 
expert  system  that  contains  the  domain  knowledge  [51:23]. 

Modus  Ponens  -  A  rule  of  inference  that  states:  If  A  then  B 
is  true,  and  A  is  true,  you  can  infer  that  B  is  true  [45:0- 
12]  . 

Natural.  l_an.g.uag.e  -  The  standard  method  of  exchanging 
information  between  people,  such  as  English  (to  be 
contrasted  with  artificial  languages,  such  as  programming 
languages)  [51:78]. 

Rule  -  A  formal  way  of  specifying  a  recommendation, 
directive,  or  strategy,  expressed  as  IF  premise  THEN 
conclusion  or  IF  condition  THEN  action  [51-23]. 

User  -  A  person  who  uses  an  expert  system,  such  as  an  end- 
user,  a  domain  expert,  a  knowledge  engineer,  a  tool  builder, 
or  a  clerical  staff  member  [51:11]. 

User  Interface  -  One.  subsystem  of  a  knowledge  system  (or  any 
computing  system)  with  which  the  human  user  deals  routinely. 
It  aims  to  be  as  "natural*  as  possible,  employing  language 
as  close  as  possible  to  ordinary  language  (or  the  stylized 
language  of  a  given  field)  and  understanding  and  displaying 
images,  all  at  speeds  that  are  comfortable  and  natural  for 
humans.  Other  subsystems  in  a  knowledge  system  are  the 
knowledge  base  and  the  inference  engine  [45:0-17], 


Appendix  B: 


Knowledge  Eng  meerini 


Exercise  One 


EXERCISE  ONE 

Please  answer  the  following  questions.  Be  as  specific 
as  possible.  Read  all  the  questions  before  answering. 

1.  Define  Asset  Reconciliation  in  general  terms. 

2.  What  do  you  consider  the  most  difficult  part  of  the 

Asset  Reconciliation  process’ 

3.  What  is  the  most  common  mistake  made  in  the  Asset 
Reconciliation  process? 

4.  Do  certain  assets  require  significantly  different 
procedures  during  Asset  Reconciliation’  If  so,  what 
types  of  assets  and  how  are  the  procedures  different’ 

5.  What  is  the  objective  of  completing  AFLC  Form  47’ 
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Appendix  C:  Knowledge  Eng ineenn<  Exercise  Two 


EXERCISE  TWO 

For  each  of  the  selected  cases,  conduct  an  asset 
reconciliation  computation  in  accordance  with  the  following 
instructions . 

Be  very  specific  and  very  detailed.  Write  out  how  you  would 
complete  the  computation  (e.g.,  first  .....  next  ....). 
Number  each  step.  Indicate  where  you  find  the  information 
required  for  the  computation. 

Please  try  to  explicitly  list  all  the  detailed  intermediate 
steps  you  go  through. 
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EXERCISE  THREE 

This  exercise  involves  role-playing.  One  of  you  will  be 
asked  to  act  as  the  expert  system  while  the  other  person 
will  act  as  the  novice  item  manager.  Imagine  a  computer 
screen  separates  you  from  the  other  person. 

The  person  acting  as  the  novice  will  attempt  to  complete  the 
selected  asset  reconciliation  case  using  the  advice  and 
suggestions  of  the  person  acting  as  the  expert  system. 

If  you  are  acting  as  the  expert  system,  please  do  not  use 
your  hands  to  point  or  gesture  in  any  way.  If  you  are 
acting  as  the  novice,  please  respond  to  the  suggestions 
offered  by  the  other  person  as  a  novice  item  manager  would. 


Appendix  E:  Testing  Facility  Requirements  and  the 
Schedule  for  Testing 


EVALUATION  REQUIREMENTS  FOR 
THE  MANAGER’S  ASSET  RECONCILIATION  SYSTEM  (MARS) 

1.  Testing  area  with  ten  IBM  compatible  computers  (Z-248s). 
There  should  be  enough  space  around  the  computers  for 
regulations,  documentation,  etc. 

2.  In  advance  of  actual  testing,  participants  should  be 

identified  and  scheduled  for  testing  on  the  22,  23,  24, 
of  June. 

a.  A  range  of  participant  experience  is  needed. 

1)  novice  -  less  than  one  year  of  experience  as  an 

item  manager 

2)  journeyman  -  one  to  four  years  of  experience  as 
an  item  manager 

3)  advanced  -  more  that  four  years  of  experience 

as  an  item  manager 

3.  Participants  should  be  scheduled  ten  at  a  time,  for  one- 
hour  and  fifteen  minute  intervals,  with  15  minute  'break 
periods  between  each  testing  session.  Participants 
should  be  told  to  bring  any  items  which  they  might 
normally  use  to  complete  a  Form  47  (regulations,  a 
caluculator,  etc.). 

4.  Option  One  (Preferred): 

Need  40  novice  item  managers. 

Need  40  journeyman  item  managers. 

Need  40  advanced  item  managers. 

Option  Two: 

Need  40  novice  item  managers. 

Need  40  'other*  item  managers. 

Other  -  more  than  one  year  of  experience  as  an  item 
manger . 

Note:  All  test  results  will  remain  anonymous. 
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Appendix  F:  Wr i tten  Directions  Provided  to  Test  Sub i ects 

DIRECTIONS  PROVIDED  TO  TEST 
SUBJECTS  WHO  WERE  ASSISTED  BY 
THE  EXPERT  SYSTEM 

The  purpose  of  this  exercise  is  to  field  test  the 
Manager’s  Asset  Reconciliation  System  (MARS).  MARS  was 
developed  to  assist  item  managers  with  the  asset 
reconciliation  computation.  This  test  is  designed  to 
determine  if  item  managers  do  a  better  job  of  asset 
reconciliation  when  assisted  by  MARS. 

You  have  been  randomly  preassigned  to  reconcile  a  case 
with  the  assistance  of  MARS.  MARS’  performance  in  advising 
you  on  the  reconciliation  of  this  case  will  be  graded  in 
accordance  with  answers  predetermined  by  a  panel  of  experts. 
The  case  you  reconcile  will  receive  a  grade  between  0  and 
100  points.  Your  MARS-assisted  performance  will  be  compared 
with  the  performance  of  item  managers  who  reconcile  this 
same  case  without  the  assistance  of  MARS. 

At  this  time,  please  complete  the  questionnaire 
entitled  SUBJECT  INFORMATION.  When  you  are  finished,  await 
further  instructions.  Do  Not  begin  the  case  until  told  to 


An  introductory  case  has  been  provided  so  that  you  may 
learn  to  use  MARS.  You  will  reconcile  this  case  first. 
MARS  will  ask  you  a  series  of  questions  about  your 
computation  and  the  supporting  documentation.  MARS  asks 
very  specific  questions.  Please  read  the  questions 
carefully  and  accurately  answer  the  questions  which  MARS 
asks  . 

You  answer  by  typing  the  correct  answer  and  depressing 
the  ENTER  key.  You  do  not  need  to  type  the  complete 
response.  Ususally  the  first  letter  of  the  response  is 
sufficient  for  MARS  to  distinguish  your  answer.  For 
example,  if  MARS  asks  you  a  yes/no  question,  you  would 
answer  yes  by  typing  'y*  and  depressing  the  ENTER  key. 

Based  upon  your  responses,  MARS  will  provide  advice  on 
the  correct  quantity  of  assets  for  your  Form  47.  If  you  have 
any  questions,  raise  your  hand.  One  of  the  test 
administrators  will  assist  you. 

Upon  completion  of  the  introductory  session,  you  will 
be  provided  the  test  case  on  which  MARS’  performance  will  be 
measured.  One  of  the  test  administrators  will  give  a  brief 
description  of  the  test  case,  then  you  may  begin.  When  you 
finish  this  case,  please  provide  your  feedback  by  completing 
the  page  entitled  MARS  CRITIQUE.  Then  you  may  leave. 
Please  leave  quietly  as  others  may  still  be  working. 


Please  comply  with  the  following  rules: 


1.  Do  not  write  in  the  case  folder.  Use  the  back  of  your 
questionnaire  if  you  need  to  make  notes. 

2.  Do  not  talk  with  anyone  about:  this  case  or  what  you 
believe  to  be  the  correct  answers. 

3.  You  may  ask  questions  only  of  a  test  administrator.  To 
do  so,  raise  your  hand. 

4.  Do  not  talk  with  anyone  else  in  the  room  or  make  any 
comments  aloud. 

5.  Do  not  worry  about  others  finishing  before  you. 
Performance  in  terms  of  the  reconciled  asset  quantity  is  the 
primary  concern.  Time  should  not  be  considered  as  a  factor. 

6.  Do  not  put  your  name  on  any  of  the  forms.  This  exercise 
is  anonymous. 

7.  Remember  to  complete  all  the  required  entries  for  the 
asset  reconciliation  computation. 

8.  A  test  administrator  will  verify  that  all  entries  on  the 
questionnaires  are  complete  before  you  leave. 

9.  Please  do  not  remove  anything  from  this  room.  If  you 
took  notes  on  the  case,  you  must  turn  them  in  to  an 
administrator  with  your  questionnaires. 
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DIRECTIONS  PROVIDED  TO  TEST 
SUBJECTS  WHO  WERE  NOT  ASSISTED 
BY  THE  EXPERT  SYSTEM 

The  purpose  of  this  exercise  is  to  field  test  the 
Manager’s  Asset  Reconci I i ati on  System  (MARS).  MARS  was 
developed  to  assist  item  managers  with  the  asset 
reconciliation  computation.  This  test  is  designed  to 
determine  if  item  managers  do  a  better  job  of  asset 
reconciliation  when  assisted  by  MARS. 

You  have  been  randomly  preassigned  to  reconcile  a  case 
without  the  assistance  of  MARS  (manually).  Your  performance 
in  reconciling  this  case  will  be  graded  in  accordance  with 
answers  predetermined  by  a  panel  of  experts.  The  case  you 
reconcile  will  receive  a  grade  between  C  and  100  points. 
Your  performance  will  be  compared  with  the  performance  of 
item  managers  who  reconcile  this  same  case  with  the 
assistance  of  MARS. 

At  this  time,  please  complete  the  questionnaire 
entitled  SUBJECT  INFORMATION.  When  you  are  finished,  await 
further  instructions.  Do  Not  begin  the  case  until  told  to 
do  so . 

You  will  be  provided  the  test  case  on  which  your 
performance  will  be  measured.  One  of  the  test 
ad  mi n i s tr ator s  will  give  a  brief  description  of  the  test 
case,  then  you  may  begin.  If  you  have  any  questions,  please 
raise  your  hand.  One  of  the  test  administrators  will  assist 


you.  When  you  finish  this  case,  please  leave  quietly  as 
others  may  still  be  working. 

Please  comply  with  the  following  rules: 

1.  Do  not  write  in  the  case  folder.  Use  the  back  of  your 
questionnaire  if  you  need  to  make  notes. 

2.  Do  not  talk  with  anyone  about  this  case  or  what  you 
believe  to  be  the  correct  answers. 

3.  You  may  ask  questions  only  of  a  test  administrator.  To 
do  so,  raise  your  hand. 

4.  Do  not  talk  with  anyone  else  in  the  room  or  make  any 
comments  aloud. 

5.  Do  not  worry  about  others  finishing  before  you. 
Performance  in  terms  of  the  reconciled  asset  quantity  is  the 
primary  concern.  Time  should  not  be  considered  as  a  factor. 

6.  Do  not  put  your  name  on  any  of  the  forms.  This  exercise 
is  anonymous. 

7.  Remember  to  complete  all  the  required  entries  for  the 
asset  reconciliation  computation. 

8.  A  test  administrator  will  verify  that  all  entries  on  the 
questionnaires  are  complete  before  you  leave. 

9.  Please  do  not  remove  anything  from  this  room.  If  you 
took  notes  on  the  case,  you  must  turn  them  in  to  an 
administrator  with  your  questionnaires. 
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SUBJECT  INFORMATION 


Appendix  G:  Ques 


Please  answer  the  following  questions: 

1.  I  have  been  a  D041  item  manager  for  _ 

2.  I  am  performing  case  number  _  . 

3.  I  am  doing  this  case  (check  one) 

_  manually 

_  with  the  expert  system 

4.  I  would  rate  my  experience  in  interactin 
computer  as:  (check  one) 

_  No  experience  at  all 

_  Limited  experience 

_  Mediocre 

_  Experienced 

_  Very  experienced 


5.  I  would  rate  my  confidence  in  interactin 
personal  computer  as:  (check  one) 

_  No  confidence 

_  Limited  confidence 

_  Mediocre 


High 


Very  high 
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years  . 
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Appendix  H;  Supporting  Documentation  f  or  the  Complex 
As3et  Reconci 1 iation  Test  Case 

15  April  87 
1610-00-071-5472 

D165  Tight  Item  Report  dated  15  Apr  87  reports  65  total 
assets  on  hand.  31  Mar  87  SB  &  CR  reported  30  assets.  Form 
47  increased  by  35.  for  Mar  87  cycle. 
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Appendix  I : 


A  SAMPLE  CONSULTATION  WITH  MARS 


Welcome  to  the 

Manager's  Asset  Reconciliation  System 

(MARS) 


This  Artificial  Intelligence  Expert  System  was  developed  by 
the  Air  Force  Institute  of  Technology  School  of  Logistics 
with  the  help  of  Materiel  Management  experts  from  Warner 
Robins  Air  Logistics  Center. 


Have  we  ever  had  the  pleasure  of  working  together  before? 
(answer  by  typing  in  yes  or  no  and  then  hit  the  carriage 
return) 

yes 

no 

>>  n 


Hi!  I'm  MARS.  My  job  is  to  assist  you  in  the  maintenance  of 
asset  reconciliation  records.  Together  we  can  provide  a 
complete,  current,  accurate  and  permanent  record  of  assets 
in  the  system.  I  will  work  with  you,  step  by  step,  through 
the  AFLC  Form  47.  Occasionally,  I  will  recommend  specific 
actions  to  you.  I  can't  provide  advice  on  every  problem. 
Sometimes  you  may  still  have  to  ask  your  supervisor. 
However,  I  can  help  you  with  most  things.  I’ll  do  my  very 
best . 


when  you  are  ready  to  continue,  type  ’c’  and  hit  enter  key. 
c 

>  >  c 
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What  I’m  going  to  do  is  ask  you  several  questions  regarding 
the  information  on  your  Form  47  and  supporting  documentation 
When  I  ask  a  question,  answer  with  one  of  the  suggested 
answers  which  follow  each  question  then  hit  the  carriage 
return . 

If  you  are  unsure  how  to  answer  a  question  type  'unknown' 
and  I  will  give  you  a  detailed  explanation  of  what  I  am 
asking . 

Now  let’s  get  started.  When  you  are  ready  to  continue,  type 
’c’  and  hit  enter  key. 
c 

>>  c 


Enter  the  National  Stock  Number  (subgroup  master)  in  quotes 
now. 

Example  ’ 4320-00-07 1 -  1 74  5PP ’  . 


any  lower-case  expression. 
>>  * 1234-11-654-3245BB’ 


Enter  the  Item  Activity  Code  (IAC)  in  quotes  now. 

Example  ’C8’.  Answer  ’unknown*  for  an  explanation  of 
activity  code  designation. 


any  lower-case  expression. 
>>  ’  C4  ' 


Enter  the  ’as  of  date’  in  now. 


Example  30/06/87. 


1.  All  procurements  in  support  of  POS ,  WRSK/BLSS,  OWRM,  FMS 
stock  levels.  This  includes  all  current  on-order 
undelivered  quantities. 

2.  ISSP. 

3.  Contract  termination. 

4.  On-order  PR  funded. 

5.  Asset  gains  from  prior  cycle. 

a.  Reclamation 

b.  Modification 

c.  Depot  IAVs 

Caution:  Do  not  include  FMS  direct  cite,  installs,  other 
service  MIPRs ,  and  unfunded  PRs/MIPRs. 

Block  1.  What  is  the  total  quantity  of  acquired  assets 
from  the  AFLC  form  318?  Include  the  above: 

If  form  318  is  i  nco  mp  1  e  t  e  /  unava  i  1  ab  1  e  ,  then  the  value  for 
Block  1  should  equal  total  reported  in  SB  &  CR  (D104)  plus 
any  verifiable  losses.  Check  the  factors  printout, 
transaction  registers,  and  SB  &  CR  Reconciliation,  etc. 

an  integer. 

>>  1 10 

total -acquired-assets  =  110  (100%)  because  you  said  so. 

Ok? 

yes 

no 

>>  y 

Did  you  exclude  FMS  direct  cite,  i ns t a 1 1 s , o t her  service 
MIPRs,  and  unfunded  PRs/MIPRs9 
yes 
no 

>>  y 


good  ! 


Block  2  should  equal  Block  1  unless: 

1.  The  previous  form  47  indicates  starting  position 
has  been  adjusted  on  a  prior  cycle. 

2.  A  decision  is  made  to  adjust  Block  2  based  on  the 
variance  for  the  three,  most  recent,  consecutive  cycles 

Enter  the  quantity  for  Block  2  now.  (i.e.  previous  qtr 
Block  2  qty  *  any  variances  ♦  any  asset  gains) 

Note:  An  opportunity  to  adjust  Block  2  for  Block  13  variance 
will  be  provided  shortly. 


an  integer  between  0  and  100000. 

>>  105 

a tar t ing -pos i t i on  =  105  (100%)  because  you  said  so. 

Ok? 

yes 

no 

>>  y 


Is  the  Block  2  quantity  different  from  the  Block  1  quantity? 

yes 

no 

>>  y 


Annotate  the  date  on  the  Form  47  that  established  Block  2. 
Example  30/06/87. 

Note:  If  you  are  going  to  adjust  for  variance  this  quarter, 
use  the  current  quarter  date. 


any  lower-case  expression. 
>>  31/03/87 


Does  block  13  of  the  Form  47  indicate  that  the  item  has 
experienced  three  consecutive  cycles  with  positive  variances 
or  three  consecutive  cycles  with  minus  variances? 


yes 

no 

>>  y 


What  is  the  smallest  absolute  value  of  the  three  consecutive 
positive  variances  or  the  three  consecutive  negative 
variances? 

Example:  If  you  had  -8,  -5,  and  -9  for  your  previous  three 
cycles;  you  would  enter  5  at  this  time. 

an  integer. 

>>  3 
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Is  the  variance  positive 
or  negative7 


positive 

negative 

>>  n 


102  is  the  new  starting  position;  explain  the  reason  for 
adjustment  in  the  remarks  block. 


Enter  your  remarks  in  quotes  now. 

Example  "Block  2  adjusted  for  variance  noted  in  the  previous 
three  cycles  .  " 

Note:  If  remarks  are  not  required,  type  n/a  and  continue. 


any  lower-case  expression. 

>>  "Block  2  adjusted  for  negative  variance." 


Enter  the  number  of  condemnations  accumulated  from  the 
previous  quarter  now. 


an  integer  between  0  and  100000. 

>>  2 


Block  3.  How  many  condemnations  occurred  this  quarter? 
You  could  check  the  factors  printout,  D032/D033  weekly, 
12  months  transaction  history  and  SB  &  CR. 


an  integer  between  0  and  100000. 

>  >  2 


Have  you  included 
condemns7  Type  "no 
line  as  necessary. 


base  condemnations  and  depot  SB  &  CR 
to  go  back  and  adjust  the  condemnations 


4  is  the  accumulated  quantity  for  condemnations  this  quarter 


Enter  the  number  of  mock-ups  accumulated  from  the  previous 
quarter  now. 

an  integer. 

>>  0 


Block  3.  How  many  mock-ups  have  occurred  this  quarter? 
Check  the  transaction  register  for  project  441  issues  to 
repair  facility.  Enter  the  total  accumulated  quantity. 

an  integer  between  0  and  100000. 

>>  0 


0  is  the  accumulated  quantity  for  mockups  this  cycle. 

Block  3.  Enter  the  number  of  installations  accumulated  from 
the  previous  quarter  now. 

an  integer. 

>>  1 


Block  3.  How  many  installations  or  shipments  for  spram  have 
occurred  this  quarter?  Check  the  SB  &  CR  for  installations 
and  the  transaction  register  for  project  code  428  issues  as 
spram. 


an  integer  between  0  and  100000. 

>>  2 


3  is  the  accumulated  quantity  for  installations  this  cycle. 

Enter  the  number  of  test  equipment  items  accumulated  from 
the  previous  quarter  now. 


M 


m 


) 


’  *4 


Block  3.  Enter  the  number  of  test  equipment  items 
accumulated  from  the  current  quarter  that  were  issued  to  a 
repair  facility  for  testing,  check  out  purposes,  or  to  fill 
a  table  of  allowance  requirement. 

an  integer  between  0  and  100000. 

>>  1 


2  is  the  accumulated  quantity  for  test  equipment  this  cycle. 

Enter  the  number  of  shipments  to  FMS  accumulated  from  the 
previous  quarter  now. 


an  integer. 

>>  0 


Block  3  (shipments  to  FMS/SA).  How  many  assets  have  been 
shipped  from  stock  this  quarter  to  FMS/SA  customers’  Check 
the  SB  &  CR  shipment  to  SAP,  then  deduct  H  coded 
requisitions  shown  in  the  transaction  register  under  the 
FB2065  quantity. 


Note:  If  using  transaction  register: 

Do  not  count  direct  shipments  from  contractors. 

Do  not  count  shipments  from  WRM  accounts. 

Do  not  count  other  services  off-base  storage  sites. 


an  integer  between  0  and  100000. 

>>  0 


0  is  the  accumulated  quantity  for  shipments  to  FMS  this  cycle. 


Enter  the  number  of  NRA  items  accumulated  from  the 
previous  quarter  now. 


Block  3,  Check  the  SB  &  CR  end/or  transaction  registers  to 
determine  how  many  assets  were  shipped  from  stock  this 
quarter  to  NRA.  Non  reporting  activities  are  listed  in  AFM 
67-1,  vol  I,  part  2,  chapter  1,  attachment  1. 

1.  Include  shipments  to  material  utilization  control  office 

2.  Do  not  include  SICA  services  when  assets  are  to  be 
returned  or  unserviceable  exchanged  for  serviceable 

3.  Do  not  include  DMISA  items. 


an  integer  between  0  and  100000. 

>>  0 


1  is  the  accumulated  quantity  for  NRA  this  cycle. 


Block  3.  How  many  assets  will  not  be  returned  to  the  AF 
spares  inventory  due  to  special  project  requisition? 

Enter  the  accumulated  total  now. 

Note:  An  availability  request  should  have  been 
submitted  for  any  special  project  losses. 


an  integer  between  0  and  100000. 

>>  2 


Block  3.  How  many  assets  were  lost  to  modifications  this 
quarter9  1.  Include  assets  modified  to  another  configuration 
and  not  part  of  this  subgrouping.  2.  Include  assets  used  in 
modification  of  a  Next  Higher  Assembly.  If  you  suspect 
losses,  check  with  production  management  or  the  equipment 
special i st . 

Enter  the  accumulated  total  now. 


an  integer  between  0  and  100000. 

>>  2 


Block  3.  Include  the  number  of  depot  Inventory 
Adjustment  Vouchers,  reported  in  the  SB  &  CR,  on  the  Minus 
I A V  line. 

Enter  the  accumulated  total  now. 

an  integer  between  0  and  100000. 

>  >  7 
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Block  3.  Look  at  the  transfers  to  the  Defense  Re¬ 
utilization  and  Marketing  Office  (DRMO)  shown  in  the  SBCR 
under  items  coded  disposal,  this  number  represents  the 
number  of  assets  which  have  been  declared  non  reparable 
without  being  condemned.  You  should  have  processed  a 
transfer  document  to  get  these  assets  shipped  to  R  &  M. 
Enter  the  accumulated  total  now. 


an  integer  between  0  and  100000. 

>>  6 


Block  3.  How  many  asset  losses  have  not  been  listed  in  any 
of  the  previous  categories?  Enter  the  accumulated  total 
now.  Examples:  Gifts  of  assets  to  the  Air  Museum,  items 
donated  to  State  or  County  governments.  Annotate  the 
accumulated  total  of  any  such  asset  losses  on  the  line 
labeled  'other"  and  explain  in  the  remarks  section  at  the 
bottom  of  the  form  47. 


an  integer  between  0  and  100000. 

>>  4 
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3. 


ASSET  LOSSEg 

a.  Condemnations  4 

b.  Mock-ups  0 

c.  Instal lations/SPRAM  3 

d.  Test  Equipment  2 

e.  Shipments  to  FMS/SA  Programs  0 

f.  Shipments  to  Non-Reporting  Activities  1 

g.  Special  Projects  2 

h.  Modification  2 

i .  Minus  I AV  7 

j .  Transfers  to  DPDO  6 

k.  Other  (explain  in  remarks)  4 


TOTAL(Asset  Losses)  31 

If  the  values  listed  above  are  not  to  your  satisfaction 
restart  the  consultation  or  continue  and  an  opportunity  to 
change  the  values  for  condemnations,  FMS/SA  program  items, 
NRA  items.  Minus  IAV  will  provided  later  in  this  program. 
When  you  are  ready  to  continue,  type  ’c’  and  hit  enter  key. 
c 

>>  c 


Block  4.  How  many  assets  were  gained  this  quarter  by  an 
inventory  adjustment?  Look  at  the  SBCR  and  disregard  base 
I AVs  using  depot  IAVs  only. 


an  integer  between  0  and  100000. 

>>  8 


Block  4.  How  many  assets  were  gained  this  quarter  through 
modification  of  an  item,  aircraft,  or  system?  Any  gain 
reported  here  should  be  the  result  of  a  decrease  in  assets 
on  the  old,  unmodified  NSN. 


an  integer  between  0  and  100000. 
>>  12 


Block  4.  How  many  other  gains  have  not  been  listed? 
Examples  include,  discontinued  special  projects  and  the 
reclamation  of  a  Next  Higher  Assembly  or  an  aircraft. 
Annotate  any  such  gains  on  the  line  labeled  ’other*  and 
explain  in  the  remarks  section  at  the  bottom  of  this  Form 
47. 

an  integer  between  0  and  100000. 

>>  9 


8 

12 

9 


4.  ASSET  GAINS 

a.  Plus  IAV 

b.  Modification 

c.  Other  (explain  in  remarks) 

TOTAL (Asset  Gains)  29 

Remember  that  asset  gains  are  non  accumulative  and  are  shown 
in  Block  4  for  the  current  quarter  only;  should  not  be 
included  in  block  1  or  2  of  the  cycle  being  reconciled. 

100  represents  your  balance 
accountable  assets  value  for  block  5. 

when  you  are  ready  to  continue,  type  ’c’  and  hit  enter  key. 
c 

>>  c 

Block  6.  How  many  funded  PRs/MIPRs  are  due  in  as  of  the 
asset  cut-off  date?  Look  at  the  master  and  subs  overlay 
from  the  J041,  you  should  have  received  a  339  card  also. 
This  information  is  also  listed  on  the  form  318. 

Note:  Do  not  include  other  service  MIPRs/PRs  or  assets  on 
contract  for  direct  FMS  shipment. 

an  integer  between  0  and  100000. 

>>  10 


If  you  had  funded  PR/MIPR,  did  you  include  this  quantity 
in  blocks  1  and  2  ? 

yes 

no 

>>  y 

Good  i 

Each  PR  should  have  been  confirmed  as  funded  by  review  of 
the  H057  report. 


What  is  the  total  quantity  on  order  contract  as  of  the 
cutoff  date?  Look  at  the  overlay  from  the  J041,  should  have 
received  a  33P  card,  also  posted  on  AFLC  Form  318. 


an  integer  between  0  and  100000. 

2 


Information  entry. 

What  is  the  total  of  all  unfunded  PRs/MIPRs?  Look  at  J041, 
include  PRs  that  were  initiated  without  money  being 
committed  as  of  the  asset  cut  off  date  of  the  computation. 

You  should  check  the  H057.2L1  accounting  document  or  special 
ALC  products,  provided  for  this  purpose,  to  determine  if  any 
unfunded  PRs  were  actually  committed  as  of  the  asset  cut  off 
date;  if  they  were,  they  should  be  included  in  the  quantity 
listed  under  *0n  order  PR/MIPR  (funded)'. 

an  integer  between  0  and  100000. 

>>12 


Did  you  include  them  in  your  on  order  PR/MIPR  (funded) 
quantity? 

yes 

no 

>>  y 


Good  ! 


Block  6.  How  many  assets  are  due  in  from  ISSP?  Look  at  the 
J041  for  any  projected  receipts  from  other  services,  do  not 
include  projected  unserviceable  returns  from  SICA  services. 


an  integer  between  0  and  100000. 

>>  5 


Block  6.  How  many  assets  are  due  in  from  reclamation?  Look 
at  the  J041,  D067,  other  applicable  records  and  include 
assets  reclaimed  from  Next  Higher  Assemblies,  crashed  or 
damaged  aircraft,  and  save  lists. 

Enter  ’why’  if  you  need  further  information. 


an  integer  between  0  and  100000. 


>>  4 


Have  you  verified  the  reclaimed  asset  value  by  reviewing 
the  following? 


1.  A-D067. -R41-R1-MR4. 

2.  AFLC  Form  206  for  reclamation  of  Next  Higher  Assembly. 

If  you  enter  no  or  unknown,  the  system  will  ask  you  to  re¬ 
enter  the  number  of  assets  due  in  from  reclamation. 


yes 

no 


Block  6.  How  many  assets  are  due  in  from  reclamation?  Look 
at  the  J041,  D067,  other  applicable  records  and  include 
assets  reclaimed  from  Next  Higher  Assemblies,  crashed  or 
damaged  aircraft,  and  save  lists. 

Enter  'why'  if  you  need  further  information. 


an  integer  between  0  and  100000. 

>>  6 


Have  you  verified  the  reclaimed  asset  value  by  reviewing 
the  following? 

1.  A-D067. -R41-R1-MR4 . 

2.  AFLC  Form  206  for  reclamation  of  Next  Higher  Assembly. 

If  you  enter  no  or  urtknown ,  the  system  will  ask  you  to  re-enter 
the  number  of  assets  due  in  from  reclamation. 


yes 

no 

>>  y 

Good  ! 


How  many  assets  are  in  contract  termination  status  but  still 
due  in?  This  information  is  advertised  by  the  D/MM  MUC0  to 
D/MM  Item  Managers  like  yourself.  You  can  submit  DD  form 
1149  to  obtain  contract  termination  quantity  information. 


an  integer  between  0  and  100000. 

>>  8 


How  many  other  due-in  assets  are  there?  An  example  would  be 
an  item  pending  modification  that  is  not  a  part  of  this 
subgroup.  Be  sure  to  explain  in  the  remarks  block  if  you 
have  any  assets  in  this  category. 

an  integer  between  0  and  100000. 

>>  0 


6.  DUE-IN  ASSETS 

a.  On  order  PR/MIPB  (Funded)  10 

b.  On  order  contract  2 

c.  Unfunded  PR  (Information  Entry)  12 

d.  ISSP  5 

e.  Reclamation  6 

f.  Contract  Termination  8 

g.  Other  (explain  in  remarks)  0 


T0TAL(Due-in  Assets)  31 


69  represents  your  net 
balance  accountable  for  block  7. 

when  you  are  ready  to  continue,  type  'c’  and  hit  enter  key. 
c 

>  >  c 

Block  8. 

What  are  your  current  assets  as  reported  in  the  SB  &  CR 
(overlay  from  D104  system  initial  comp)? 

The  following  entries  are  optional  entries  listed  under 
block  8: 

Serviceables  both  base  and  depot,  include  DIFM 

Unserviceables  both  base  and  depot 

Intransit  serviceable 

Intransit  unserviceable 

WRM  serviceable 

Due-In  from  Overhaul  (DI0H) 

Other  (explain  in  remarks) 

The  total  of  all  the  entries,  if  entered,  should  be 
reflected  in  the  initial  D041  requirements  computation. 

an  integer  between  0  and  100000. 

>>  62 


Block  9.  What  is  the  verified  DOTM  quantity  to  include  any 
"H"  coded  FMS  requisitions?  Note:  You  should  verify  DOTM  by 
calling  applicable  base  if  DOTM  quantity  is  suspect. 


Reference  WRALC-MMOI  57-4,  Appendix  1,  pg  6  for  a  more 
detailed  explanation  of  DOTM. 


an  integer  between  0  and  100000. 

>>  3 
Ok? 


yes 

no 

>>  V 


59  is  the  net  SB  &  CR  assets  quantity  to  be  entered  in 
Block  10. 


-10  is  the  difference  value 

to  be  entered  in  Block  11  and  represents  an  underage  of 
items  reported  this  quarter. 


This  is  a  suggested  area  to  research  in  order  to  resolve 
your  difference  value 

Have  you  verified  the  correct  value  for  condemnations  by 
accomplishing  the  following?  Review  of  the  contractor’s  end 
item  report  (form  413)  Review  of  the  D032/D033  weekly 
transaction  register  Contacting  PMS 

yes 

no 

>>  n 

The  condemnation  quantity  you  have  computed  for  this 
reconciliation  is  =  4  (100%)  because  rule-152. 


If  you  wish  to  change  the  value  for  condemnations,  enter  the 
amount  of  change  (plus  or  minus)  now.  Example  20  or  -20. 

If  you  do  not  wish  to  change  the  condemnations  value,  enter 

0. 


8  is  the  adjuster 

condemnations  value  to  be  entered  on  the  Form  47 


35  is  the  adjusted 

total  asset  losses  value  to  be  entered  on  the  Form  47 

96  is  the  adjusted 

balance  accountable  assets  value 

65  is  the  adjusted 

net  balance  accountable  value 

-6  is  the  adjusted 
difference  value  for  Block  11 

when  you  are  ready  to  continue,  type  'c'  and  hit  enter  key. 
c 

>>  c 


Have  you  verified  the  number  of  shipments  to  FMS/SA 
programs,  as  reported  in  the  SB  &  CR,  by  review  of  the 
D032/D033  weekly  transaction  register? 


yes 

no 

>>  y 


Excellent ! 


Have  the  following  items  been  excluded  from  your 
shipments  to  Non-Reporting-Activities: 

1.  shipments  to  other  services  for  DMISMA  repair 

2.  shipments  to  contractors  for  depot  overhaul  repair 
of  programmed  depot  maintenance 


yes 

no 


Outstanding ! 


v  V  v  .v  V  V 


Have  the  minus  IAV  quantities  been  verified  by  the  base 
supply  IM  and/or  review  of  the  D032/D033  weekly  transaction 
register? 


>  > 


yes 

no 

y 


Keep  up  the  good  work! 


Have  the  plus  IAV  quantities  been  verified  by  the  base 
supply  IM  and/or  review  of  the  D032/D033  weekly  transaction 
register"? 


yes 

no 


t  Good  going! 

i 

l 

i  Have  the  on-order  contract  quantities  been  verified  by 

|  review  of  the  due-in  asset  card  (Form  339)  in  conjunction 

1  with  a  D008  12  months  transaction  history  or  the  D032/D033 

I  weekly  transaction  register? 


yes 

no 

;  >>  y 

I 

i 

,  Great ! 


•For  your  information  only*  Block  12  is  used  to  adjust 
reported  assets  after  appropriate  research  is  accomplished 
to  determine  the  validity  of  the  assets  reported  in  the  SB  & 
CR.  Research  all  available  records  such  as  inventory 
records,  reports  histories,  and  transaction  histories.  You 
should  enter  only  those  quantities  with  known  causes  that 
can  be  explained. 

Strike  any  key  to  continue. 
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any  lower-case  expression 


>>  c 


How  many  assets  were  determined,  after  research,  to  have 
been  reported  incorrectly  by  using  activities  (i.e. 
contractors,  AF  bases,  etc.)  ? 


an  integer. 

>>  1 

1  j  s  the  value  to  be  entered  in  the 

the  adjustments  to  reported  assets  (Block  12) . 

-5  is  the  net  difference  value  to  be  entered  in  Block  13 
and  denotes  variance  that  cannot  be  explained. 

-7.69231%  is  the  percentage  net  difference  value  to  be 
entered  in  Block  14.  Note:  I  AW  AFLCB  57-4,  Worldwide 
Inventory  required  for  the  following: 

1.  Variance  of  ♦/-  5%  equaling  *500,000  or  more  for 
buy,  termination,  or  repair  items. 

2.  Variance  of  +/-  10%  equaling  *1,000,000  or  more  for 
items  not  computing  buy,  repair,  or  termination. 

68  is  the  on-hand  assets  value  to  be  used  in  the  final 
computation  (Block  15).  If  you  cannot  reconcile  after 
research,  explain  in  the  remarks  section  that  this  quantity 
is  the  sum  of  net  balance  accountable  and  DOTM. 

When  you  are  ready  to  continue,  type  ’c’  and  hit  enter  key. 
c 

>>  c 
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Appendix  J :  Sample  Output  from  a  MARS  Consul  tat i on 


ASSET  RECONCILIATION  INVESTMENT  ITEMS 


NATIONAL  STOCK  NUMBER 
ITEM  ACTIVITY  CODE 
AS  OF  DATE 


1234-11-854- 3245BB 

C4 

30/9/87 


1.  TOTAL  ACQUIRED  ASSETS 

2.  STARTING  POSITION  ( Date : ) 3 1 /3/87 

3.  ASSET  LOSSES 
Condemnation* 

Mock-up* 

I natal lations/SPRAM 
Teat  Equipment 

Shipments  to  FMS/SA  Program* 

Shipment*  to  Non-Reporting  Activities 
Special  Project* 

Modi f ication 
Minus  IAV 
Transfer*  to  DPDO 
Other  (explain  in  remarks) 


102 


1 10 


TOTALIAsset  Losses)  31 

ASSET  GAINS 

a.  Plus  IAV  8 

b.  Modification  12 

c.  Other  (explain  in  remarks)  9 


BALANCE  ACCOUNTABLE  ASSETS 


TOTAL (Asset  Gains)  29 

100 


DUE- IN  ASSETS 

On  order  PR/MIPR  (Funded) 

On  order  contract 
Unfunded  PR  (Information  Entry) 
ISSP 

Rec 1 amat i on 
Contract  Termination 
Other  (explain  in  remarks) 


10 

2 

12 

5 

6 
8 

0 


TOTAL ( Due -in  Assets)  31 

69 


NET  BALANCE  ACCOUNTABLE 
CURRENT  SB  A  CR  ASSETS  65 

DOTM  (Included  in  Block  B)  3 

NET  SB  A  CR  ASSETS  (Block  8  -  Block  9)  62 

DIFFERENCE  (Block  7  vs  Block  10)  -7 

ADJUSTMENT  TO  REPORTED  ASSETS  AFTER  RESEARCH  2 

NET  DIFFERENCE  (Block  11  Block  12)  -5 

PERCENTAGE  OF  NET  DIFFERENCE  (Block  13/Block7)  -7.24638X 
ON  HAND  ASSETS  JSED  IN  COMPUTATION  72 

REMARKS:  Block  2  adjusted  lot  negative  variance. 


1  66 


Appendix  K: 


MARS  CRITIQUE 


Please  use  the  space  below  to  make  any  comments  about  MARS 
(e.g.,  what  you  liked  or  did  not  like  about  the  system,  if 
it  helped  you,  or  how  it  might  be  improved) . 


Appendix  L:  Test  Subject  Critiques  o  f  the 
Manager ' s  Asset  Re cone  1 1 lation  System 


TEST  SUBJECT  CRITIQUES  OF  THE 
MANAGER’S  ASSET  RECONCILIATION  SYSTEM 


In  a  word  GREAT1  MARS  teaches  while  the  IM  accomplishes  the 
task  of  reconciliation.  Would  like  to  have  scroll  back  to 
adjusts  something  when  I  need  to  One  point:  There  were  two 
computer  illiterate  people  (and  there  are  many  at  this  ALC) 
in  the  class  May  I  suggest  that  you  scan  your  class  and 
look  for  the  older  members 

I  believe  it  will  be  an  excellent  too;  in  working  Form  4"’  s 
once  you  get  adjusted  to  it  Yes  it  helped  me 

I  liked  the  questions  listed.  it  reminds  you  to  double 
check 

It  helped  clarify  some  additives 
Need  more  training 

I  think  this  will  be  very  beneficia* 

This  would  help  me  a  lot  Once  I  get  a  little  more 
experience  in  D041  But  overall  I  liked  it 

Questions  asked  clear  and  good,  problem  with  correcting 
errors 

Set  up  and  answers  were  good  I  didn't  like  the  inability 
to  roll  back  wh enever  you  wa nted  to 

Very  good  potential  If  and  when  we  get  to  use  ir 

With  a  direct  interface  between  the  SB  &  CR  and  the  J  0  4  < 
systems,  the  MARS  system  would  save  us  alot  of  time  It 
would  be  helpful  if  the  information  from  the  previous  qtrs 
such  as  starting  position  and  the  variances  in  block  13 
could  be  displayed 

I  think  the  system  will  be  a  good  asset  to  the  IM 
I  endorse  the  MARS  concept 

I  feel  the  MARS  program  will  be  extremely  helpful  to  the 
1 tem  manager 
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I  like  the  system. 


|  This  will  enhance  the  47  system  greatly.  My  hats  off  H 

i  Life  saver ' 

i 

i 

|  A  great  program.  Additive  qty  was  distracting. 

I  like  the  computer  doing  the  calculations  for  the  IM.  I 
would  like  to  see  the  system  operate  with  the  numbers  in  the 
|  fields  only  and  the  procedures  and  other  info  on  a  help 

screen 

Very  good  program.  Only  change  suggested  would  be  to  put 
most  of  the  explanations  for  the  different  blocks  under  a 
'help'  screen.  Experienced  persons  can  eliminate  most  of 
the  read  1 ng . 

Great  idea.  Good  teach i n*  tool.  We  need  this  kind  of  thing 
in  other  areas  This  will  be  of  more  help  to  the  IM  when 
the  system  will  accumulate  quantities  and  carry  forward 
quantities  from  one  quarter  to  another. 

Being  new  in  0041  item  management,  this  system  seemed  very 
helpful  in  that  it  walks  through  each  step  of  the  47  and 
explains  where  to  obtain  information  in  order  to  complete 
the  form  I  hope  that  it  will  be  used  in  the  future. 

Wording  needs  to  be  improved  but  shows  great  potential. 

After  several  training  sessions  would  be  much  quicker  than 
filling  in  Form,  Program  is  good. 

!  This  provides  more  details  than  the  manual  procedure.  This 

i s  a  he  1 p . 

j  -lack  of  familiarity  with  this  program  caused  me  to  make  a 

I  key  punch  error  when  I  tried  to  speed  up  my  entries. 

'  -this  program  is  user  friendly  and  easy  to  pick  up  --  with 

experience  it  would  be  nice  to  be  able  to  skip  unused 
|  elements  and  go  directly  to  the  next  entry  element. 

i 

It  is  a  good  system  and  after  the  IM's  have  had  more  time  to 

l  become  more  familiar,  I  am  sure  will  become  a  valuable  tool. 

! 

j  Recommend  each  block  be  visible  on  the  screen  so  IM  can  view 

this  all  the  time  to  make  changes  if  necessary. 

i  I  think  that  MARS  is  trulyl  helpful  and  I  feel  that  the 

|  sooner  it  is  implemented  into  the  system,  the  better. 


Excellent  program. 


Would  like  to  have  this  program  go  into  effect  immediately 
Can  be  of  invaluable  assistance  to  all  IM’s! 


This  program  is  helpful  especially  to  an  inexperienced  IM  in 
preparing  an  AFLC  Form  47  and  can  be  used  by  all  to  refresh 
your  memory  for  this  task.  The  only  problem  I  found  was  an 
incompatibility  with  MARS  terminology  for  line  12  vs 
terminology  indicated  on  AFLC  Form  47. 

The  system  should  really  make  IM's  better  informed  and  more 
accurate  when  doing  form  47 ' s . 

I  feel  that  once  this  product  is  implemented  into  system 
will  tremendously  enhance  the  reconciliation  process.  What 
I  saw  in  this  short  orientation  was  impressive. 

This  will  be  a  very,  very  helpful  tool  in  D041.  I  enjoyed 
learning  something  that  will  help  me  in  my  job.  Thank  you. 

I  have  only  been  an  IM  about  a  month,  however,  I  thought  the 
system  was  good.  I  did  work  a  few  47’s,  and  this  process 
helped  me  to  understand  what  I  was  really  doing. 

I  think  this  system  offers  clear,  concise  questions 
concerning  Form  47’s  -  simple  enough.  It  should  help. 

Goo d  idea  to  eliminate  time  involved  in  manually  working 
Form  47’s.  I  enjoyed  it! 

I  liked  the  extra  information  you  could  get  on  what  to 
consider  and  not  consider  on  arriving  at  info  input.  I  had 
a  little  trouble  going  back  to  correct  errors. 

Very  good  system.  Would  like  to  do  more.  Very  challenging 
structure.  This  is  what  is  needed  to  learn  how  to  do  these 
things . 

The  questions  seemed  very  good  and  thorough  for  the 
information  given.  The  biggest  problem  I  had  was  not 
understanding  where  to  pick-up  the  information  from  off  the 
sheets.  I’m  not  a  seasoned  IM. 

In  the  future  this  will  be  a  great  time  saver. 

Liked  'MARS’  all  right.  I  am  a  new  IM  and  not  an  expert  on 
47’s,  so  I  got  sorta  lost.  There  needs  to  be  more  of  a 
mechanism  to  change  earlier  entries.  Other  than  that,  it 
was  quite  user  friendly. 
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Looks  good!  I  like  this.  Will  be  more  interesting  to  work 
the  47’s  this  way.  Should  save  time  looking  through  so  many 
regs  and  flipping  through  so  many  sheets  if  you  could  get 
the  NSN  info  in  the  system. 
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